[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:04:04 dist-test-slave-fjww krb5kdc[6919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:04:04 dist-test-slave-fjww krb5kdc[6919](info): set up 2 sockets
Sep 28 03:04:04 dist-test-slave-fjww krb5kdc[6919](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 03:04:05.816792 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:04:05 dist-test-slave-fjww krb5kdc[6919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:04:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:04:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:04:06.016477 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:46633
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:46633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:06.361189 6932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:06.361613 6932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:06.361802 6932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:06.369738 6932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:04:06.369952 6932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:06.370081 6932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:06.370198 6932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:04:06.370288 6932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:04:06.381181 6932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:46633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:46633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:06.382709 6932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:06.384254 6932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:06.397737 6938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:06.397769 6937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:06.397729 6940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:06.398837 6932 server_base.cc:1034] running on GCE node
I20260928 03:04:06.399858 6932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:06.401463 6932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:06.402788 6932 hybrid_clock.cc:648] HybridClock initialized: now 1790564646402727 us; error 47 us; skew 500 ppm
I20260928 03:04:06.403221 6932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:06 dist-test-slave-fjww krb5kdc[6919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564646, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:06.407130 6932 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:04:06.408936 6932 webserver.cc:460] Webserver started at http://127.6.192.62:36321/ using document root <none> and password file <none>
I20260928 03:04:06.409637 6932 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:06.409744 6932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:06.410018 6932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:06.413352 6932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "547f4d414e9945429303f284070c55c0"
format_stamp: "Formatted at 2026-09-28 03:04:06 on dist-test-slave-fjww"
server_key: "91b544c260891cd507f68cf78e9288a1"
server_key_iv: "d77e14eb897533b5ceb0c3e3358f5a3f"
server_key_version: "encryptionkey@0"
I20260928 03:04:06.414285 6932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "547f4d414e9945429303f284070c55c0"
format_stamp: "Formatted at 2026-09-28 03:04:06 on dist-test-slave-fjww"
server_key: "91b544c260891cd507f68cf78e9288a1"
server_key_iv: "d77e14eb897533b5ceb0c3e3358f5a3f"
server_key_version: "encryptionkey@0"
I20260928 03:04:06.419881 6932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 03:04:06.423820 6947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:06.425443 6932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:04:06.425663 6932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "547f4d414e9945429303f284070c55c0"
format_stamp: "Formatted at 2026-09-28 03:04:06 on dist-test-slave-fjww"
server_key: "91b544c260891cd507f68cf78e9288a1"
server_key_iv: "d77e14eb897533b5ceb0c3e3358f5a3f"
server_key_version: "encryptionkey@0"
I20260928 03:04:06.425959 6932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:06.438958 6932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:06.444288 6932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:06.444656 6932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:06.464872 6932 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:46633
I20260928 03:04:06.464872 6998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:46633 every 8 connection(s)
I20260928 03:04:06.466429 6932 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:04:06.470640 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:06.475997 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 6932
I20260928 03:04:06.476321 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:04:06.478317 6912 external_mini_cluster.cc:1428] Setting key bb9f6ee84aa336ff2ddca6dda4b8a28b
I20260928 03:04:06.481974 6999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0: Bootstrap starting.
I20260928 03:04:06.485550 6999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:06.486672 6999 log.cc:824] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:06.489791 6999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0: No bootstrap required, opened a new log
I20260928 03:04:06.500061 6999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547f4d414e9945429303f284070c55c0" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46633 } }
I20260928 03:04:06.500625 6999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:06.500756 6999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547f4d414e9945429303f284070c55c0, State: Initialized, Role: FOLLOWER
I20260928 03:04:06.501639 6999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547f4d414e9945429303f284070c55c0" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46633 } }
I20260928 03:04:06.501962 6999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:04:06.502113 6999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:04:06.502332 6999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:06.504954 6999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547f4d414e9945429303f284070c55c0" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46633 } }
I20260928 03:04:06.505595 6999 leader_election.cc:302] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 547f4d414e9945429303f284070c55c0; no voters:
I20260928 03:04:06.506364 6999 leader_election.cc:288] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:04:06.506629 7004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:06.508302 7004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0 [term 1 LEADER]: Becoming Leader. State: Replica: 547f4d414e9945429303f284070c55c0, State: Running, Role: LEADER
I20260928 03:04:06.509020 7004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547f4d414e9945429303f284070c55c0" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46633 } }
I20260928 03:04:06.509321 6999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:04:06.512495 7005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "547f4d414e9945429303f284070c55c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547f4d414e9945429303f284070c55c0" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46633 } } }
I20260928 03:04:06.512941 7005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0 [sys.catalog]: This master's current role is: LEADER
Sep 28 03:04:06 dist-test-slave-fjww krb5kdc[6919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:06.516633 7006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 547f4d414e9945429303f284070c55c0. Latest consensus state: current_term: 1 leader_uuid: "547f4d414e9945429303f284070c55c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547f4d414e9945429303f284070c55c0" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46633 } } }
I20260928 03:04:06.517079 7006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:06.518831 7009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:04:06.524955 7009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:04:06.533847 7001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:06.482585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53488 (local address 127.6.192.62:46633)
0928 03:04:06.483627 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:06.483650 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:06.485083 (+ 1433us) server_negotiation.cc:407] Connection header received
0928 03:04:06.495224 (+ 10141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:06.495262 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:06.495995 (+ 733us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:06.496541 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:06.500569 (+ 4028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:06.502979 (+ 2410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:06.505230 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:06.506552 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:06.506812 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:06.517492 (+ 10680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:06.517511 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:06.517539 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:06.517616 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:06.523941 (+ 6325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:06.525657 (+ 1716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:06.525670 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:06.525682 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:06.526063 (+ 381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:06.526966 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:06.526974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:06.526980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:06.528013 (+ 1033us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:06.531158 (+ 3145us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:06.531633 (+ 475us) server_negotiation.cc:299] Negotiation successful
0928 03:04:06.532146 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":463,"threads_started":1}
I20260928 03:04:06.543509 7009 catalog_manager.cc:1347] Generated new cluster ID: 6dec677b55d44fc893761471c127ac53
I20260928 03:04:06.543635 7009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:04:06.565866 7009 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:04:06.567305 7009 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:04:06.575745 7009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 547f4d414e9945429303f284070c55c0: Generated new TSK 0
I20260928 03:04:06.576629 7009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
I20260928 03:04:06.629772 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:46633
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:07.025349 7025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:07.025717 7025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:07.025914 7025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:07.034492 7025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:07.034691 7025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:07.034895 7025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:04:07.046300 7025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:46633
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:07.047813 7025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:07.049319 7025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:07.063912 7031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:07.063863 7033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:07.063963 7030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:07.065091 7025 server_base.cc:1034] running on GCE node
I20260928 03:04:07.065779 7025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:07.066815 7025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:07.068114 7025 hybrid_clock.cc:648] HybridClock initialized: now 1790564647068063 us; error 73 us; skew 500 ppm
I20260928 03:04:07.068523 7025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:07 dist-test-slave-fjww krb5kdc[6919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564647, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:07.076568 7025 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:04:07.078341 7025 webserver.cc:460] Webserver started at http://127.6.192.1:39433/ using document root <none> and password file <none>
I20260928 03:04:07.079118 7025 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:07.079265 7025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:07.079665 7025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:07.083000 7025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "510cf35395df429b842ab00c490f164d"
format_stamp: "Formatted at 2026-09-28 03:04:07 on dist-test-slave-fjww"
server_key: "e0330290291c7d2c4490ee222c59d064"
server_key_iv: "822f948b7fa62d8da3d1a8deccaec7f8"
server_key_version: "encryptionkey@0"
I20260928 03:04:07.083866 7025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "510cf35395df429b842ab00c490f164d"
format_stamp: "Formatted at 2026-09-28 03:04:07 on dist-test-slave-fjww"
server_key: "e0330290291c7d2c4490ee222c59d064"
server_key_iv: "822f948b7fa62d8da3d1a8deccaec7f8"
server_key_version: "encryptionkey@0"
I20260928 03:04:07.089608 7025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:04:07.094134 7040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:07.095913 7025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 03:04:07.096127 7025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "510cf35395df429b842ab00c490f164d"
format_stamp: "Formatted at 2026-09-28 03:04:07 on dist-test-slave-fjww"
server_key: "e0330290291c7d2c4490ee222c59d064"
server_key_iv: "822f948b7fa62d8da3d1a8deccaec7f8"
server_key_version: "encryptionkey@0"
I20260928 03:04:07.096468 7025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:07.132089 7025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:07.136901 7025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:07.137233 7025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:07.138305 7025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:07.139985 7025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:07.140091 7025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:07.140257 7025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:07.140336 7025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:07.176230 7025 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:37665
I20260928 03:04:07.176294 7152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:37665 every 8 connection(s)
I20260928 03:04:07.177584 7025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:04:07.182749 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 7025
I20260928 03:04:07.183040 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:04:07.183728 6912 external_mini_cluster.cc:1428] Setting key ca1928ba033657066ebac4080673fa4e
Sep 28 03:04:07 dist-test-slave-fjww krb5kdc[6919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564647, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:07.199570 7155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:07.180515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:53779 (local address 127.6.192.62:46633)
0928 03:04:07.180999 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:07.181004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:07.181950 (+ 946us) server_negotiation.cc:407] Connection header received
0928 03:04:07.184245 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:07.184251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:07.184350 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:07.184477 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:07.186325 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:07.187066 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:07.188817 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:07.189229 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:07.189367 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:07.193558 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:07.193565 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:07.193569 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:07.193612 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:07.196327 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:07.197103 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:07.197109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:07.197113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:07.197239 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:07.197712 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:07.197717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:07.197722 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:07.198022 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:07.198185 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:07.198913 (+ 728us) server_negotiation.cc:299] Negotiation successful
0928 03:04:07.199155 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":233,"threads_started":1}
WARNING: no policy specified for kudu/127.6.192.2@KRBTEST.COM; defaulting to no policy
I20260928 03:04:07.200733 7156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:07.181090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:46633 (local address 127.6.192.1:53779)
0928 03:04:07.181775 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:07.181822 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:07.183954 (+ 2132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:07.184783 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:07.184797 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:07.185557 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:07.186152 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:07.186170 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:07.187230 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:07.187242 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:07.188643 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:07.188655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:07.189469 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:07.189478 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:07.189736 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:07.190449 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:07.190478 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:07.193312 (+ 2834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:07.196487 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:07.196497 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:07.196516 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:07.196959 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:07.197355 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:07.197359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:07.197363 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:07.197588 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:07.198156 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:07.198164 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:07.198538 (+ 374us) client_negotiation.cc:769] Sending connection context
0928 03:04:07.198817 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 03:04:07.199387 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":279,"threads_started":1}
I20260928 03:04:07.202678 7153 heartbeater.cc:347] Connected to a master server at 127.6.192.62:46633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.2@KRBTEST.COM" created.
I20260928 03:04:07.203039 7153 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:07.204109 7153 heartbeater.cc:511] Master 127.6.192.62:46633 requested a full tablet report, sending...
I20260928 03:04:07.206883 6964 ts_manager.cc:194] Registered new tserver with Master: 510cf35395df429b842ab00c490f164d (127.6.192.1:37665)
I20260928 03:04:07.208580 6964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:53779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
I20260928 03:04:07.227667 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:46633
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:07.568362 7159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:07.568742 7159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:07.568922 7159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:07.576792 7159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:07.576963 7159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:07.577142 7159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:04:07.587251 7159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:46633
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:07.588804 7159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:07.590276 7159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:07.605458 7164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:07.605362 7165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:07.605432 7167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:07.611236 7159 server_base.cc:1034] running on GCE node
I20260928 03:04:07.612120 7159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:07.613258 7159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:07.614494 7159 hybrid_clock.cc:648] HybridClock initialized: now 1790564647614455 us; error 43 us; skew 500 ppm
I20260928 03:04:07.614899 7159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:07 dist-test-slave-fjww krb5kdc[6919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564647, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:07.618536 7159 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:04:07.620371 7159 webserver.cc:460] Webserver started at http://127.6.192.2:33999/ using document root <none> and password file <none>
I20260928 03:04:07.621165 7159 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:07.621317 7159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:07.621662 7159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:07.624809 7159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "eea8104936ec4cd0bb1ad2f3679dc148"
format_stamp: "Formatted at 2026-09-28 03:04:07 on dist-test-slave-fjww"
server_key: "06cb0fab1e91fa526f05a7852afe2039"
server_key_iv: "e949859bfc1d73d9d4b838ac9bb6c516"
server_key_version: "encryptionkey@0"
I20260928 03:04:07.625639 7159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "eea8104936ec4cd0bb1ad2f3679dc148"
format_stamp: "Formatted at 2026-09-28 03:04:07 on dist-test-slave-fjww"
server_key: "06cb0fab1e91fa526f05a7852afe2039"
server_key_iv: "e949859bfc1d73d9d4b838ac9bb6c516"
server_key_version: "encryptionkey@0"
I20260928 03:04:07.631244 7159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 03:04:07.635154 7174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:07.636680 7159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:04:07.636945 7159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "eea8104936ec4cd0bb1ad2f3679dc148"
format_stamp: "Formatted at 2026-09-28 03:04:07 on dist-test-slave-fjww"
server_key: "06cb0fab1e91fa526f05a7852afe2039"
server_key_iv: "e949859bfc1d73d9d4b838ac9bb6c516"
server_key_version: "encryptionkey@0"
I20260928 03:04:07.637183 7159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:07.668030 7159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:07.673756 7159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:07.674139 7159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:07.675309 7159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:07.676968 7159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:07.677088 7159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:07.677248 7159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:07.677342 7159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:07.716169 7159 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:45637
I20260928 03:04:07.716218 7286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:45637 every 8 connection(s)
I20260928 03:04:07.717599 7159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:04:07.727062 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 7159
I20260928 03:04:07.727361 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:04:07.727980 6912 external_mini_cluster.cc:1428] Setting key 2ce1258134bbd078452f8daf00d40a13
Sep 28 03:04:07 dist-test-slave-fjww krb5kdc[6919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564647, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:07.743955 7289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:07.720774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:40939 (local address 127.6.192.62:46633)
0928 03:04:07.721302 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:07.721309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:07.722420 (+ 1111us) server_negotiation.cc:407] Connection header received
0928 03:04:07.724496 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:07.724502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:07.724582 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:07.724695 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:07.726483 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:07.727355 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:07.729643 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:07.730510 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:07.730671 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:07.735243 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:07.735251 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:07.735259 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:07.735311 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:07.739589 (+ 4278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:07.740585 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:07.740594 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:07.740600 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:07.740814 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:07.741486 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:07.741495 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:07.741501 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:07.741951 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:07.742126 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:07.743103 (+ 977us) server_negotiation.cc:299] Negotiation successful
0928 03:04:07.743434 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":245,"threads_started":1}
I20260928 03:04:07.744951 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:07.721417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:46633 (local address 127.6.192.2:40939)
0928 03:04:07.722169 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:07.722214 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:07.724134 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:07.724990 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:07.725000 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:07.725643 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:07.726277 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:07.726296 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:07.727568 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:07.727576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:07.729496 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:07.729513 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:07.730810 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:07.730821 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:07.731160 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:07.732217 (+ 1057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:07.732245 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:07.734740 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:07.739823 (+ 5083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:07.739832 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:07.739849 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:07.740392 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:07.741033 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:07.741037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:07.741039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:07.741343 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:07.742143 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:07.742150 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:07.742627 (+ 477us) client_negotiation.cc:769] Sending connection context
0928 03:04:07.742978 (+ 351us) client_negotiation.cc:240] Negotiation successful
0928 03:04:07.743644 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":318,"threads_started":1}
WARNING: no policy specified for kudu/127.6.192.3@KRBTEST.COM; defaulting to no policy
I20260928 03:04:07.747009 7287 heartbeater.cc:347] Connected to a master server at 127.6.192.62:46633
I20260928 03:04:07.747478 7287 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:07.748270 7287 heartbeater.cc:511] Master 127.6.192.62:46633 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.3@KRBTEST.COM" created.
I20260928 03:04:07.750159 6964 ts_manager.cc:194] Registered new tserver with Master: eea8104936ec4cd0bb1ad2f3679dc148 (127.6.192.2:45637)
I20260928 03:04:07.751621 6964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:40939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
I20260928 03:04:07.775521 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:46633
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:08.117102 7293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:08.117419 7293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:08.117519 7293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:08.125115 7293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:08.125254 7293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:08.125380 7293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:04:08.136302 7293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:46633
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:08.137691 7293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:08.139096 7293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:08.152971 7299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:08.153059 7301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:08.152997 7298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:08.153838 7293 server_base.cc:1034] running on GCE node
I20260928 03:04:08.154474 7293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:08.155535 7293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:08.156781 7293 hybrid_clock.cc:648] HybridClock initialized: now 1790564648156729 us; error 46 us; skew 500 ppm
I20260928 03:04:08.157140 7293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:08 dist-test-slave-fjww krb5kdc[6919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564648, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:08.160720 7293 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:04:08.162405 7293 webserver.cc:460] Webserver started at http://127.6.192.3:45225/ using document root <none> and password file <none>
I20260928 03:04:08.163139 7293 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:08.163292 7293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:08.163683 7293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:08.166738 7293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "d24e8b4ac1384a968e730ebc4abbc891"
format_stamp: "Formatted at 2026-09-28 03:04:08 on dist-test-slave-fjww"
server_key: "66e30f76a117afb1bec694cc4f2c0885"
server_key_iv: "80e1367454ea0250160c53d91eb23919"
server_key_version: "encryptionkey@0"
I20260928 03:04:08.167603 7293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "d24e8b4ac1384a968e730ebc4abbc891"
format_stamp: "Formatted at 2026-09-28 03:04:08 on dist-test-slave-fjww"
server_key: "66e30f76a117afb1bec694cc4f2c0885"
server_key_iv: "80e1367454ea0250160c53d91eb23919"
server_key_version: "encryptionkey@0"
I20260928 03:04:08.173147 7293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 03:04:08.177314 7308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:08.178948 7293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 03:04:08.179167 7293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "d24e8b4ac1384a968e730ebc4abbc891"
format_stamp: "Formatted at 2026-09-28 03:04:08 on dist-test-slave-fjww"
server_key: "66e30f76a117afb1bec694cc4f2c0885"
server_key_iv: "80e1367454ea0250160c53d91eb23919"
server_key_version: "encryptionkey@0"
I20260928 03:04:08.179525 7293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:08.204010 7293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:08.208660 7293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:08.209028 7293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:08.210124 7293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:08.211858 7293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:08.211970 7293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:08.212129 7293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:08.212205 7293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:08.212613 7153 heartbeater.cc:503] Master 127.6.192.62:46633 was elected leader, sending a full tablet report...
I20260928 03:04:08.249346 7293 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:45459
I20260928 03:04:08.249408 7420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:45459 every 8 connection(s)
I20260928 03:04:08.250809 7293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:04:08.254261 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 7293
I20260928 03:04:08.254536 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:04:08.255095 6912 external_mini_cluster.cc:1428] Setting key 4cc9255c8b3d859b94ecbee6650622af
Sep 28 03:04:08 dist-test-slave-fjww krb5kdc[6919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564648, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:08.274483 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:08.253892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:58539 (local address 127.6.192.62:46633)
0928 03:04:08.254468 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:08.254475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:08.255692 (+ 1217us) server_negotiation.cc:407] Connection header received
0928 03:04:08.258459 (+ 2767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:08.258465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:08.258578 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:08.258704 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:08.260790 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:08.261586 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:08.263280 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:08.263760 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:08.263868 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:08.267654 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:08.267659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:08.267663 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:08.267701 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:08.270652 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:08.271763 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:08.271767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:08.271769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:08.271989 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:08.272588 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:08.272593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:08.272596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:08.273051 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:08.273194 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:08.274007 (+ 813us) server_negotiation.cc:299] Negotiation successful
0928 03:04:08.274194 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":256,"threads_started":1}
I20260928 03:04:08.275712 7424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:08.254456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:46633 (local address 127.6.192.3:58539)
0928 03:04:08.255470 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:08.255525 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:08.257995 (+ 2470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:08.258979 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:08.258989 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:08.259903 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:08.260581 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:08.260608 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:08.261772 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:08.261777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:08.263127 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:08.263141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:08.263986 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:08.263991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:08.264199 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:08.264954 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:08.264979 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:08.267288 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:08.270827 (+ 3539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:08.270837 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:08.270858 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:08.271620 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:08.272144 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:08.272149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:08.272152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:08.272436 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:08.273229 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:08.273236 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:08.273610 (+ 374us) client_negotiation.cc:769] Sending connection context
0928 03:04:08.273884 (+ 274us) client_negotiation.cc:240] Negotiation successful
0928 03:04:08.274425 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":380,"threads_started":1}
I20260928 03:04:08.277293 7421 heartbeater.cc:347] Connected to a master server at 127.6.192.62:46633
I20260928 03:04:08.277711 7421 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:08.278501 7421 heartbeater.cc:511] Master 127.6.192.62:46633 requested a full tablet report, sending...
I20260928 03:04:08.280103 6964 ts_manager.cc:194] Registered new tserver with Master: d24e8b4ac1384a968e730ebc4abbc891 (127.6.192.3:45459)
I20260928 03:04:08.280962 6964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:58539
I20260928 03:04:08.286055 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 03:04:08 dist-test-slave-fjww krb5kdc[6919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564648, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 03:04:08 dist-test-slave-fjww krb5kdc[6919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564648, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:04:08.335743 7428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:08.318833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57976 (local address 127.6.192.1:37665)
0928 03:04:08.319627 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:08.319634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:08.320091 (+ 457us) server_negotiation.cc:407] Connection header received
0928 03:04:08.320237 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:08.320245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:08.320572 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:08.320719 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:08.322170 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:08.323191 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:08.324547 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:08.325148 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:08.325252 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:08.329428 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:08.329443 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:08.329458 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:08.329501 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:08.332735 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:08.333480 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:08.333488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:08.333497 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:08.333644 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:08.334049 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:08.334053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:08.334055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:08.334529 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:08.334788 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:08.335021 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 03:04:08.335241 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":315,"threads_started":1}
W20260928 03:04:08.337059 7068 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57976
Sep 28 03:04:08 dist-test-slave-fjww krb5kdc[6919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 03:04:08 dist-test-slave-fjww krb5kdc[6919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:04:08.385990 7428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:08.369620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57984 (local address 127.6.192.1:37665)
0928 03:04:08.369884 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:08.369891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:08.370055 (+ 164us) server_negotiation.cc:407] Connection header received
0928 03:04:08.370199 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:08.370203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:08.370360 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:08.370465 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:08.372024 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:08.372760 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:08.374046 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:08.374582 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:08.374694 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:08.379750 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:08.379756 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:08.379759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:08.379802 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:08.382693 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:08.383762 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:08.383766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:08.383768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:08.383929 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:08.384484 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:08.384487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:08.384490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:08.384801 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:08.385021 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:08.385366 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 03:04:08.385649 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 03:04:08.387641 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 7025
I20260928 03:04:08.401633 7428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:08.388645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57992 (local address 127.6.192.1:37665)
0928 03:04:08.388807 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:08.388810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:08.389250 (+ 440us) server_negotiation.cc:407] Connection header received
0928 03:04:08.389425 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:08.389429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:08.389490 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:08.389580 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:08.390890 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:08.391713 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:08.393053 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:08.393471 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:08.393564 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:08.396138 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:08.396142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:08.396146 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:08.396180 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:08.398841 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:08.399576 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:08.399581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:08.399583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:08.399732 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:08.400119 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:08.400123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:08.400125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:08.400432 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:08.400639 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:08.401040 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 03:04:08.401273 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 03:04:08.402402 7148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:08.491586 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 7025
I20260928 03:04:08.507668 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 7159
Sep 28 03:04:08 dist-test-slave-fjww krb5kdc[6919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.2@KRBTEST.COM
I20260928 03:04:08.525278 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:08.508534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38376 (local address 127.6.192.2:45637)
0928 03:04:08.509053 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:08.509060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:08.509086 (+ 26us) server_negotiation.cc:407] Connection header received
0928 03:04:08.509144 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:08.509151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:08.509358 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:08.509529 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:08.510845 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:08.511956 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:08.513261 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:08.513822 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:08.513976 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:08.518364 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:08.518378 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:08.518394 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:08.518436 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:08.521734 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:08.522693 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:08.522703 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:08.522705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:08.522855 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:08.523359 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:08.523367 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:08.523370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:08.523837 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:08.524193 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:08.524517 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 03:04:08.524826 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":243,"threads_started":1}
I20260928 03:04:08.526675 7282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:08.611987 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 7159
I20260928 03:04:08.627879 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 7293
Sep 28 03:04:08 dist-test-slave-fjww krb5kdc[6919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.3@KRBTEST.COM
I20260928 03:04:08.645654 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:08.628923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59632 (local address 127.6.192.3:45459)
0928 03:04:08.629543 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:08.629550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:08.629580 (+ 30us) server_negotiation.cc:407] Connection header received
0928 03:04:08.629645 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:08.629651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:08.629921 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:08.630066 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:08.631536 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:08.632608 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:08.634247 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:08.634828 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:08.634945 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:08.639563 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:08.639578 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:08.639594 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:08.639634 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:08.642560 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:08.643386 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:08.643395 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:08.643397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:08.643621 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:08.644047 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:08.644051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:08.644053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:08.644468 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:08.644710 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:08.645053 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 03:04:08.645264 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":287,"threads_started":1}
I20260928 03:04:08.647149 7416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:08.734484 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 7293
I20260928 03:04:08.750353 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 6932
I20260928 03:04:08.751708 6994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:08.859212 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 6932
2026-09-28T03:04:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4180 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:04:08 dist-test-slave-fjww krb5kdc[7443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:04:08 dist-test-slave-fjww krb5kdc[7443](info): set up 2 sockets
Sep 28 03:04:08 dist-test-slave-fjww krb5kdc[7443](info): commencing operation
krb5kdc: starting...
W20260928 03:04:09.952106 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:04:10 dist-test-slave-fjww krb5kdc[7443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:04:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:04:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:04:10.152431 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:40141
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:38663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:40141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:10.492681 7456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:10.493098 7456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:10.493286 7456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:10.501871 7456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:04:10.502040 7456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:10.502163 7456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:10.502296 7456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:04:10.502381 7456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:04:10.513989 7456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:40141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:40141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:10.515527 7456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:10.516975 7456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:10.530053 7461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:10.530148 7464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:10.530161 7462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:10.531461 7456 server_base.cc:1034] running on GCE node
I20260928 03:04:10.532471 7456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:10.534152 7456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:10.535681 7456 hybrid_clock.cc:648] HybridClock initialized: now 1790564650535426 us; error 47 us; skew 500 ppm
I20260928 03:04:10.536216 7456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:10 dist-test-slave-fjww krb5kdc[7443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564650, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:10.540071 7456 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:04:10.542146 7456 webserver.cc:460] Webserver started at http://127.6.192.62:41493/ using document root <none> and password file <none>
I20260928 03:04:10.542899 7456 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:10.543052 7456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:10.543394 7456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:10.546540 7456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "6e75f18a845a4576a1f301659c250f77"
format_stamp: "Formatted at 2026-09-28 03:04:10 on dist-test-slave-fjww"
server_key: "f307d1ac6fd7f7bf300ac031904831f8"
server_key_iv: "1ab4cd17198eac69a1d3f34508e06d22"
server_key_version: "encryptionkey@0"
I20260928 03:04:10.547346 7456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "6e75f18a845a4576a1f301659c250f77"
format_stamp: "Formatted at 2026-09-28 03:04:10 on dist-test-slave-fjww"
server_key: "f307d1ac6fd7f7bf300ac031904831f8"
server_key_iv: "1ab4cd17198eac69a1d3f34508e06d22"
server_key_version: "encryptionkey@0"
I20260928 03:04:10.553108 7456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 03:04:10.557335 7471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:10.559019 7456 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 03:04:10.559355 7456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "6e75f18a845a4576a1f301659c250f77"
format_stamp: "Formatted at 2026-09-28 03:04:10 on dist-test-slave-fjww"
server_key: "f307d1ac6fd7f7bf300ac031904831f8"
server_key_iv: "1ab4cd17198eac69a1d3f34508e06d22"
server_key_version: "encryptionkey@0"
I20260928 03:04:10.559685 7456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:10.578243 7456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:10.583246 7456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:10.583649 7456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:10.605475 7456 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:40141
I20260928 03:04:10.605520 7522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:40141 every 8 connection(s)
I20260928 03:04:10.607113 7456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:04:10.611763 7523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:10.612443 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 7456
I20260928 03:04:10.612692 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:04:10.613682 6912 external_mini_cluster.cc:1428] Setting key d92dfb8645fddd951a20ea1bba621bd2
I20260928 03:04:10.627357 7523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77: Bootstrap starting.
Sep 28 03:04:10 dist-test-slave-fjww krb5kdc[7443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:10.631102 7523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:10.632256 7523 log.cc:824] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:10.635426 7523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77: No bootstrap required, opened a new log
I20260928 03:04:10.641040 7526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:10.616110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.6.192.62:40141)
0928 03:04:10.617029 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:10.617042 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:10.617084 (+ 42us) server_negotiation.cc:407] Connection header received
0928 03:04:10.619894 (+ 2810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:10.619929 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:10.620473 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:10.620981 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:10.622766 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:10.624482 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:10.625972 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:10.626794 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:10.627033 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:10.632212 (+ 5179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:10.632222 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:10.632239 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:10.632301 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:10.636052 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:10.636929 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:10.636941 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:10.636949 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:10.637164 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:10.637617 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:10.637624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:10.637634 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:10.638294 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:10.638678 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:10.639262 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 03:04:10.639796 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":687,"thread_start_us":423,"threads_started":1}
I20260928 03:04:10.643828 7523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e75f18a845a4576a1f301659c250f77" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40141 } }
I20260928 03:04:10.644272 7523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:10.644410 7523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e75f18a845a4576a1f301659c250f77, State: Initialized, Role: FOLLOWER
I20260928 03:04:10.645071 7523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e75f18a845a4576a1f301659c250f77" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40141 } }
I20260928 03:04:10.645381 7523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:04:10.645494 7523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:04:10.645614 7523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:10.648224 7523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e75f18a845a4576a1f301659c250f77" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40141 } }
I20260928 03:04:10.648746 7523 leader_election.cc:302] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e75f18a845a4576a1f301659c250f77; no voters:
I20260928 03:04:10.649291 7523 leader_election.cc:288] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:04:10.649469 7528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:10.650192 7528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77 [term 1 LEADER]: Becoming Leader. State: Replica: 6e75f18a845a4576a1f301659c250f77, State: Running, Role: LEADER
I20260928 03:04:10.650774 7528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e75f18a845a4576a1f301659c250f77" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40141 } }
I20260928 03:04:10.651643 7523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:04:10.653620 7529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e75f18a845a4576a1f301659c250f77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e75f18a845a4576a1f301659c250f77" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40141 } } }
I20260928 03:04:10.654181 7529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:10.654207 7530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e75f18a845a4576a1f301659c250f77. Latest consensus state: current_term: 1 leader_uuid: "6e75f18a845a4576a1f301659c250f77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e75f18a845a4576a1f301659c250f77" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40141 } } }
I20260928 03:04:10.654628 7530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:10.656886 7533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:04:10.662441 7533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:04:10.672586 7533 catalog_manager.cc:1347] Generated new cluster ID: a89aa91d0f164981a753a46780ed294e
I20260928 03:04:10.672732 7533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:04:10.705536 7533 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:04:10.707008 7533 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:04:10.714757 7533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e75f18a845a4576a1f301659c250f77: Generated new TSK 0
I20260928 03:04:10.715915 7533 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
I20260928 03:04:10.767982 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:40141
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:38663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:11.127523 7549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:11.127859 7549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:11.128068 7549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:11.136629 7549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:11.136806 7549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:11.137019 7549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:04:11.148977 7549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:40141
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:11.150482 7549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:11.152235 7549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:11.168170 7555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:11.168143 7554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:11.168519 7557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:11.169301 7549 server_base.cc:1034] running on GCE node
I20260928 03:04:11.170032 7549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:11.171154 7549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:11.172413 7549 hybrid_clock.cc:648] HybridClock initialized: now 1790564651172363 us; error 77 us; skew 500 ppm
I20260928 03:04:11.172784 7549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:11 dist-test-slave-fjww krb5kdc[7443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564651, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:11.176469 7549 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:04:11.178211 7549 webserver.cc:460] Webserver started at http://127.6.192.1:33087/ using document root <none> and password file <none>
I20260928 03:04:11.178969 7549 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:11.179085 7549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:11.179380 7549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:11.182750 7549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "bc6b842a983a4805968921ef013e3a7b"
format_stamp: "Formatted at 2026-09-28 03:04:11 on dist-test-slave-fjww"
server_key: "3f08c0878ae54a2e83047d5a3e9ec6ec"
server_key_iv: "b385f0e57daa51a4be778b4c06bbb228"
server_key_version: "encryptionkey@0"
I20260928 03:04:11.183732 7549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "bc6b842a983a4805968921ef013e3a7b"
format_stamp: "Formatted at 2026-09-28 03:04:11 on dist-test-slave-fjww"
server_key: "3f08c0878ae54a2e83047d5a3e9ec6ec"
server_key_iv: "b385f0e57daa51a4be778b4c06bbb228"
server_key_version: "encryptionkey@0"
I20260928 03:04:11.189595 7549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:04:11.193756 7564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:11.195322 7549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 03:04:11.195618 7549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "bc6b842a983a4805968921ef013e3a7b"
format_stamp: "Formatted at 2026-09-28 03:04:11 on dist-test-slave-fjww"
server_key: "3f08c0878ae54a2e83047d5a3e9ec6ec"
server_key_iv: "b385f0e57daa51a4be778b4c06bbb228"
server_key_version: "encryptionkey@0"
I20260928 03:04:11.195835 7549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:11.220180 7549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:11.225463 7549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:11.225730 7549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:11.226779 7549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:11.228261 7549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:11.228341 7549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:11.228430 7549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:11.228473 7549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:11.264434 7549 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:40415
I20260928 03:04:11.264467 7676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:40415 every 8 connection(s)
I20260928 03:04:11.265913 7549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:04:11.267693 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 7549
I20260928 03:04:11.267921 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:04:11.268749 6912 external_mini_cluster.cc:1428] Setting key 1522eaada0cf6004a92e577014b4ecc6
Sep 28 03:04:11 dist-test-slave-fjww krb5kdc[7443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564651, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.192.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.2@KRBTEST.COM" created.
I20260928 03:04:11.290385 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:11.268948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:36311 (local address 127.6.192.62:40141)
0928 03:04:11.269448 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:11.269455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:11.270876 (+ 1421us) server_negotiation.cc:407] Connection header received
0928 03:04:11.273001 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:11.273007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:11.273101 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:11.273220 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:11.274903 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:11.275693 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:11.277349 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:11.277769 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:11.277912 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:11.282462 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:11.282476 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:11.282483 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:11.282545 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:11.286576 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:11.287705 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:11.287710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:11.287712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:11.287857 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:11.288432 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:11.288436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:11.288438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:11.288820 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:11.288965 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:11.289919 (+ 954us) server_negotiation.cc:299] Negotiation successful
0928 03:04:11.290132 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":240,"threads_started":1}
I20260928 03:04:11.292106 7680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:11.269764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:40141 (local address 127.6.192.1:36311)
0928 03:04:11.270675 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:11.270743 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:11.272693 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:11.273488 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:11.273499 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:11.274102 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:11.274706 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:11.274724 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:11.275869 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:11.275873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:11.277179 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:11.277193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:11.278024 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:11.278031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:11.278290 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:11.279023 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:11.279048 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:11.282014 (+ 2966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:11.286808 (+ 4794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:11.286818 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:11.286835 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:11.287533 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:11.288015 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:11.288019 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:11.288021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:11.288268 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:11.288949 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:11.288956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:11.289315 (+ 359us) client_negotiation.cc:769] Sending connection context
0928 03:04:11.289745 (+ 430us) client_negotiation.cc:240] Negotiation successful
0928 03:04:11.290408 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":324,"threads_started":1}
I20260928 03:04:11.293896 7677 heartbeater.cc:347] Connected to a master server at 127.6.192.62:40141
I20260928 03:04:11.294250 7677 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:11.295038 7677 heartbeater.cc:511] Master 127.6.192.62:40141 requested a full tablet report, sending...
I20260928 03:04:11.297632 7488 ts_manager.cc:194] Registered new tserver with Master: bc6b842a983a4805968921ef013e3a7b (127.6.192.1:40415)
I20260928 03:04:11.299295 7488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:36311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
I20260928 03:04:11.315958 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:40141
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:38663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:11.655803 7683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:11.656136 7683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:11.656443 7683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:11.664431 7683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:11.664559 7683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:11.664736 7683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:04:11.675113 7683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:40141
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:11.676586 7683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:11.677927 7683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:11.693378 7691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:11.693382 7688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:11.693382 7689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:11.693893 7683 server_base.cc:1034] running on GCE node
I20260928 03:04:11.694896 7683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:11.696065 7683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:11.697337 7683 hybrid_clock.cc:648] HybridClock initialized: now 1790564651697254 us; error 73 us; skew 500 ppm
I20260928 03:04:11.697739 7683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:11 dist-test-slave-fjww krb5kdc[7443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564651, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:11.701313 7683 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:04:11.702989 7683 webserver.cc:460] Webserver started at http://127.6.192.2:46249/ using document root <none> and password file <none>
I20260928 03:04:11.703996 7683 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:11.704133 7683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:11.704488 7683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:11.707644 7683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "4757283204f046669da4e6f478fa5369"
format_stamp: "Formatted at 2026-09-28 03:04:11 on dist-test-slave-fjww"
server_key: "231615165315cdef44e1489e038f1ed7"
server_key_iv: "479d7f0da10238956000f5360d764b16"
server_key_version: "encryptionkey@0"
I20260928 03:04:11.708478 7683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "4757283204f046669da4e6f478fa5369"
format_stamp: "Formatted at 2026-09-28 03:04:11 on dist-test-slave-fjww"
server_key: "231615165315cdef44e1489e038f1ed7"
server_key_iv: "479d7f0da10238956000f5360d764b16"
server_key_version: "encryptionkey@0"
I20260928 03:04:11.714182 7683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 03:04:11.718258 7698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:11.720129 7683 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:04:11.720329 7683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "4757283204f046669da4e6f478fa5369"
format_stamp: "Formatted at 2026-09-28 03:04:11 on dist-test-slave-fjww"
server_key: "231615165315cdef44e1489e038f1ed7"
server_key_iv: "479d7f0da10238956000f5360d764b16"
server_key_version: "encryptionkey@0"
I20260928 03:04:11.720556 7683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:11.749338 7683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:11.754137 7683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:11.754489 7683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:11.755627 7683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:11.757138 7683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:11.757253 7683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:11.757421 7683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:11.757498 7683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:11.794436 7683 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:41693
I20260928 03:04:11.794490 7810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:41693 every 8 connection(s)
I20260928 03:04:11.795950 7683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:04:11.806166 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 7683
I20260928 03:04:11.806470 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:04:11.806932 6912 external_mini_cluster.cc:1428] Setting key 093c3f3c793fe7c56ecb62b429a534fd
Sep 28 03:04:11 dist-test-slave-fjww krb5kdc[7443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564651, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:11.820101 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:11.798852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:48211 (local address 127.6.192.62:40141)
0928 03:04:11.799418 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:11.799427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:11.800524 (+ 1097us) server_negotiation.cc:407] Connection header received
0928 03:04:11.802519 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:11.802524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:11.802600 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:11.802705 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:11.804522 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:11.805275 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:11.807316 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:11.807988 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:11.808139 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:11.812939 (+ 4800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:11.812949 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:11.812957 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:11.813014 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:11.816157 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:11.817337 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:11.817342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:11.817344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:11.817506 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:11.818083 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:11.818086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:11.818088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:11.818366 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:11.818512 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:11.819605 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0928 03:04:11.819811 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":272,"threads_started":1}
I20260928 03:04:11.821852 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:11.799521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:40141 (local address 127.6.192.2:48211)
0928 03:04:11.800329 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:11.800369 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:11.802214 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:11.802942 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:11.802951 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:11.803636 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:11.804292 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:11.804312 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:11.805470 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:11.805474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:11.807113 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:11.807135 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:11.808265 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:11.808273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:11.808621 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:11.809505 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:11.809535 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:11.812597 (+ 3062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:11.816361 (+ 3764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:11.816375 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:11.816400 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:11.817173 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:11.817636 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:11.817644 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:11.817651 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:11.817959 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:11.818515 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:11.818527 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:11.819062 (+ 535us) client_negotiation.cc:769] Sending connection context
0928 03:04:11.819406 (+ 344us) client_negotiation.cc:240] Negotiation successful
0928 03:04:11.820128 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":286,"threads_started":1}
I20260928 03:04:11.823732 7811 heartbeater.cc:347] Connected to a master server at 127.6.192.62:40141
I20260928 03:04:11.824162 7811 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:11.825188 7811 heartbeater.cc:511] Master 127.6.192.62:40141 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.192.3@KRBTEST.COM; defaulting to no policy
I20260928 03:04:11.827217 7488 ts_manager.cc:194] Registered new tserver with Master: 4757283204f046669da4e6f478fa5369 (127.6.192.2:41693)
I20260928 03:04:11.828162 7488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:48211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
I20260928 03:04:11.855029 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:40141
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:38663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:12.194407 7817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:12.194720 7817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:12.194828 7817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:12.202770 7817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:12.202895 7817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:12.203016 7817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:04:12.214071 7817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:40141
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:12.215404 7817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:12.216948 7817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:12.231664 7822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:12.231595 7823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:12.231639 7825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:12.232368 7817 server_base.cc:1034] running on GCE node
I20260928 03:04:12.233117 7817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:12.234189 7817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:12.235526 7817 hybrid_clock.cc:648] HybridClock initialized: now 1790564652235411 us; error 37 us; skew 500 ppm
I20260928 03:04:12.235947 7817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:12 dist-test-slave-fjww krb5kdc[7443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564652, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:12.244830 7817 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:04:12.246686 7817 webserver.cc:460] Webserver started at http://127.6.192.3:39567/ using document root <none> and password file <none>
I20260928 03:04:12.247552 7817 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:12.247709 7817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:12.248035 7817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:12.251106 7817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "b8166bb0ebc64efb8f92f7a41b3e40f7"
format_stamp: "Formatted at 2026-09-28 03:04:12 on dist-test-slave-fjww"
server_key: "e4f1c2bdfc2156b3bcdc79087e3ebc7f"
server_key_iv: "2eb4cb079aa03a264ac5e1146539735e"
server_key_version: "encryptionkey@0"
I20260928 03:04:12.252115 7817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "b8166bb0ebc64efb8f92f7a41b3e40f7"
format_stamp: "Formatted at 2026-09-28 03:04:12 on dist-test-slave-fjww"
server_key: "e4f1c2bdfc2156b3bcdc79087e3ebc7f"
server_key_iv: "2eb4cb079aa03a264ac5e1146539735e"
server_key_version: "encryptionkey@0"
I20260928 03:04:12.258191 7817 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260928 03:04:12.262388 7832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:12.264235 7817 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:04:12.264461 7817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "b8166bb0ebc64efb8f92f7a41b3e40f7"
format_stamp: "Formatted at 2026-09-28 03:04:12 on dist-test-slave-fjww"
server_key: "e4f1c2bdfc2156b3bcdc79087e3ebc7f"
server_key_iv: "2eb4cb079aa03a264ac5e1146539735e"
server_key_version: "encryptionkey@0"
I20260928 03:04:12.264755 7817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:12.279887 7817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:12.285377 7817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:12.285724 7817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:12.287235 7817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:12.288779 7817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:12.288900 7817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:12.289057 7817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:12.289136 7817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:12.302902 7677 heartbeater.cc:503] Master 127.6.192.62:40141 was elected leader, sending a full tablet report...
I20260928 03:04:12.325639 7817 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:44153
I20260928 03:04:12.325783 7944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:44153 every 8 connection(s)
I20260928 03:04:12.327037 7817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:04:12.335175 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 7817
I20260928 03:04:12.335566 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:04:12.336045 6912 external_mini_cluster.cc:1428] Setting key cedbe897d60b7c9996f6532254149655
Sep 28 03:04:12 dist-test-slave-fjww krb5kdc[7443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564652, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:12.349766 7947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.330039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:42917 (local address 127.6.192.62:40141)
0928 03:04:12.330546 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:12.330551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:12.331734 (+ 1183us) server_negotiation.cc:407] Connection header received
0928 03:04:12.333645 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:12.333651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:12.333763 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:12.333892 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:12.335654 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.336541 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.338538 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.339062 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.339192 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:12.343229 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:12.343235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:12.343238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:12.343276 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:12.346361 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:12.347286 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:12.347290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:12.347293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:12.347501 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:12.348054 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:12.348057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:12.348059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:12.348347 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:12.348496 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:12.349265 (+ 769us) server_negotiation.cc:299] Negotiation successful
0928 03:04:12.349472 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":288,"threads_started":1}
I20260928 03:04:12.351218 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.330544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:40141 (local address 127.6.192.3:42917)
0928 03:04:12.331399 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:12.331594 (+ 195us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:12.333346 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:12.334154 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:12.334163 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:12.334732 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:12.335417 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:12.335480 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.336713 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.336721 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:12.338392 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:12.338404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.339303 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.339312 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:12.339671 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:12.340439 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:12.340464 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:12.342965 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:12.346553 (+ 3588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:12.346561 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:12.346577 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:12.347134 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:12.347664 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:12.347677 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:12.347679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:12.347938 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:12.348506 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:12.348513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:12.348881 (+ 368us) client_negotiation.cc:769] Sending connection context
0928 03:04:12.349144 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 03:04:12.349710 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":337,"threads_started":1}
I20260928 03:04:12.352931 7945 heartbeater.cc:347] Connected to a master server at 127.6.192.62:40141
I20260928 03:04:12.353307 7945 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:12.354214 7945 heartbeater.cc:511] Master 127.6.192.62:40141 requested a full tablet report, sending...
I20260928 03:04:12.355948 7488 ts_manager.cc:194] Registered new tserver with Master: b8166bb0ebc64efb8f92f7a41b3e40f7 (127.6.192.3:44153)
I20260928 03:04:12.357017 7488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:42917
I20260928 03:04:12.365203 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:04:12.385785 7947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.372035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41318 (local address 127.6.192.62:40141)
0928 03:04:12.372214 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:12.372218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:12.372849 (+ 631us) server_negotiation.cc:407] Connection header received
0928 03:04:12.373069 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:12.373074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:12.373155 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:12.373267 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:12.374810 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.375619 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.376941 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.377358 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.377503 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:12.380095 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:12.380100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:12.380103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:12.380140 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:12.382803 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:12.383664 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:12.383670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:12.383673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:12.383819 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:12.384234 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:12.384239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:12.384243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:12.384593 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:12.384874 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:12.385220 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 03:04:12.385464 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 03:04:12.392988 7488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:04:12.401203 7488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:04:12.424583 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.415156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57756 (local address 127.6.192.2:41693)
0928 03:04:12.415858 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:12.415867 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:12.415898 (+ 31us) server_negotiation.cc:407] Connection header received
0928 03:04:12.416161 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:12.416170 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:12.416447 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:12.416674 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:12.417610 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.419211 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.421694 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.423228 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.423376 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:12.423645 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:12.424045 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 03:04:12.424227 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":417,"threads_started":1}
I20260928 03:04:12.425700 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.414938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:41693 (local address 127.0.0.1:57756)
0928 03:04:12.415755 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:12.415811 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:12.415959 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:12.416963 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:12.416970 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:12.417012 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:12.417386 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:12.417402 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.419384 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.419388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:12.421496 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:12.421510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.423528 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.423536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:12.423743 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:12.423806 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 03:04:12.423975 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 03:04:12.424200 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":573,"threads_started":1}
I20260928 03:04:12.426414 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.416334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:44153 (local address 127.0.0.1:44534)
0928 03:04:12.416929 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:12.416954 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:12.417102 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:12.418395 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:12.418405 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:12.418438 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:12.418955 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:12.418967 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.421119 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.421125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:12.423476 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:12.423491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.425443 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.425450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:12.425628 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:12.425670 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 03:04:12.425749 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 03:04:12.425906 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":25,"thread_start_us":281,"threads_started":1}
I20260928 03:04:12.427250 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.417309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34510 (local address 127.6.192.1:40415)
0928 03:04:12.418083 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:12.418091 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:12.418695 (+ 604us) server_negotiation.cc:407] Connection header received
0928 03:04:12.419029 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:12.419038 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:12.419265 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:12.419489 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:12.420257 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.421686 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.424096 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.425685 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.425879 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:12.426107 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:12.426760 (+ 653us) server_negotiation.cc:299] Negotiation successful
0928 03:04:12.426934 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":437,"threads_started":1}
I20260928 03:04:12.427646 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.416643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.6.192.3:44153)
0928 03:04:12.417726 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:12.417735 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:12.417774 (+ 39us) server_negotiation.cc:407] Connection header received
0928 03:04:12.417888 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:12.417895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:12.418153 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:12.419146 (+ 993us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:12.419471 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.420974 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.423671 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.425190 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.426776 (+ 1586us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:12.427024 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:12.427084 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 03:04:12.427281 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":908,"thread_start_us":766,"threads_started":1}
I20260928 03:04:12.428021 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.417615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:40415 (local address 127.0.0.1:34510)
0928 03:04:12.418581 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:12.418603 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:12.418814 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:12.419597 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:12.419607 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:12.419633 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:12.420051 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:12.420062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.421856 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.421861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:12.423926 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:12.423942 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.426369 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.426378 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:12.426552 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:12.426583 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 03:04:12.427527 (+ 944us) client_negotiation.cc:240] Negotiation successful
0928 03:04:12.427650 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":54,"thread_start_us":649,"threads_started":1}
I20260928 03:04:12.430356 7746 tablet_service.cc:1467] Processing CreateTablet for tablet e5be4c5f218e48b78351dfdec03abfb6 (DEFAULT_TABLE table=test-table [id=829523518e5340e68fc402d91ba474bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:12.431073 7880 tablet_service.cc:1467] Processing CreateTablet for tablet e5be4c5f218e48b78351dfdec03abfb6 (DEFAULT_TABLE table=test-table [id=829523518e5340e68fc402d91ba474bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:12.432065 7746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5be4c5f218e48b78351dfdec03abfb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:12.431700 7612 tablet_service.cc:1467] Processing CreateTablet for tablet e5be4c5f218e48b78351dfdec03abfb6 (DEFAULT_TABLE table=test-table [id=829523518e5340e68fc402d91ba474bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:12.432497 7880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5be4c5f218e48b78351dfdec03abfb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:12.433051 7612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5be4c5f218e48b78351dfdec03abfb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:12.445053 7964 tablet_bootstrap.cc:492] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7: Bootstrap starting.
I20260928 03:04:12.447913 7965 tablet_bootstrap.cc:492] T e5be4c5f218e48b78351dfdec03abfb6 P 4757283204f046669da4e6f478fa5369: Bootstrap starting.
I20260928 03:04:12.449750 7966 tablet_bootstrap.cc:492] T e5be4c5f218e48b78351dfdec03abfb6 P bc6b842a983a4805968921ef013e3a7b: Bootstrap starting.
I20260928 03:04:12.450208 7964 tablet_bootstrap.cc:654] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:12.451895 7965 tablet_bootstrap.cc:654] T e5be4c5f218e48b78351dfdec03abfb6 P 4757283204f046669da4e6f478fa5369: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:12.452037 7964 log.cc:824] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:12.453194 7965 log.cc:824] T e5be4c5f218e48b78351dfdec03abfb6 P 4757283204f046669da4e6f478fa5369: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:12.453297 7966 tablet_bootstrap.cc:654] T e5be4c5f218e48b78351dfdec03abfb6 P bc6b842a983a4805968921ef013e3a7b: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:12.454658 7966 log.cc:824] T e5be4c5f218e48b78351dfdec03abfb6 P bc6b842a983a4805968921ef013e3a7b: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:12.455403 7964 tablet_bootstrap.cc:492] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7: No bootstrap required, opened a new log
I20260928 03:04:12.455739 7964 ts_tablet_manager.cc:1397] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 03:04:12.456593 7965 tablet_bootstrap.cc:492] T e5be4c5f218e48b78351dfdec03abfb6 P 4757283204f046669da4e6f478fa5369: No bootstrap required, opened a new log
I20260928 03:04:12.456879 7965 ts_tablet_manager.cc:1397] T e5be4c5f218e48b78351dfdec03abfb6 P 4757283204f046669da4e6f478fa5369: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260928 03:04:12.457666 7966 tablet_bootstrap.cc:492] T e5be4c5f218e48b78351dfdec03abfb6 P bc6b842a983a4805968921ef013e3a7b: No bootstrap required, opened a new log
I20260928 03:04:12.457929 7966 ts_tablet_manager.cc:1397] T e5be4c5f218e48b78351dfdec03abfb6 P bc6b842a983a4805968921ef013e3a7b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 03:04:12.466100 7966 raft_consensus.cc:348] T e5be4c5f218e48b78351dfdec03abfb6 P bc6b842a983a4805968921ef013e3a7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4757283204f046669da4e6f478fa5369" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41693 } } peers { permanent_uuid: "b8166bb0ebc64efb8f92f7a41b3e40f7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44153 } } peers { permanent_uuid: "bc6b842a983a4805968921ef013e3a7b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 40415 } }
I20260928 03:04:12.466562 7966 raft_consensus.cc:374] T e5be4c5f218e48b78351dfdec03abfb6 P bc6b842a983a4805968921ef013e3a7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:12.466687 7966 raft_consensus.cc:729] T e5be4c5f218e48b78351dfdec03abfb6 P bc6b842a983a4805968921ef013e3a7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc6b842a983a4805968921ef013e3a7b, State: Initialized, Role: FOLLOWER
I20260928 03:04:12.467531 7966 consensus_queue.cc:260] T e5be4c5f218e48b78351dfdec03abfb6 P bc6b842a983a4805968921ef013e3a7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4757283204f046669da4e6f478fa5369" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41693 } } peers { permanent_uuid: "b8166bb0ebc64efb8f92f7a41b3e40f7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44153 } } peers { permanent_uuid: "bc6b842a983a4805968921ef013e3a7b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 40415 } }
I20260928 03:04:12.468119 7964 raft_consensus.cc:348] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4757283204f046669da4e6f478fa5369" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41693 } } peers { permanent_uuid: "b8166bb0ebc64efb8f92f7a41b3e40f7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44153 } } peers { permanent_uuid: "bc6b842a983a4805968921ef013e3a7b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 40415 } }
I20260928 03:04:12.468119 7965 raft_consensus.cc:348] T e5be4c5f218e48b78351dfdec03abfb6 P 4757283204f046669da4e6f478fa5369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4757283204f046669da4e6f478fa5369" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41693 } } peers { permanent_uuid: "b8166bb0ebc64efb8f92f7a41b3e40f7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44153 } } peers { permanent_uuid: "bc6b842a983a4805968921ef013e3a7b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 40415 } }
I20260928 03:04:12.468658 7965 raft_consensus.cc:374] T e5be4c5f218e48b78351dfdec03abfb6 P 4757283204f046669da4e6f478fa5369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:12.468755 7964 raft_consensus.cc:374] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:12.468863 7964 raft_consensus.cc:729] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8166bb0ebc64efb8f92f7a41b3e40f7, State: Initialized, Role: FOLLOWER
I20260928 03:04:12.468890 7965 raft_consensus.cc:729] T e5be4c5f218e48b78351dfdec03abfb6 P 4757283204f046669da4e6f478fa5369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4757283204f046669da4e6f478fa5369, State: Initialized, Role: FOLLOWER
I20260928 03:04:12.469563 7966 ts_tablet_manager.cc:1428] T e5be4c5f218e48b78351dfdec03abfb6 P bc6b842a983a4805968921ef013e3a7b: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260928 03:04:12.469657 7964 consensus_queue.cc:260] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4757283204f046669da4e6f478fa5369" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41693 } } peers { permanent_uuid: "b8166bb0ebc64efb8f92f7a41b3e40f7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44153 } } peers { permanent_uuid: "bc6b842a983a4805968921ef013e3a7b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 40415 } }
I20260928 03:04:12.469712 7965 consensus_queue.cc:260] T e5be4c5f218e48b78351dfdec03abfb6 P 4757283204f046669da4e6f478fa5369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4757283204f046669da4e6f478fa5369" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41693 } } peers { permanent_uuid: "b8166bb0ebc64efb8f92f7a41b3e40f7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44153 } } peers { permanent_uuid: "bc6b842a983a4805968921ef013e3a7b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 40415 } }
I20260928 03:04:12.472213 7811 heartbeater.cc:503] Master 127.6.192.62:40141 was elected leader, sending a full tablet report...
I20260928 03:04:12.473474 7965 ts_tablet_manager.cc:1428] T e5be4c5f218e48b78351dfdec03abfb6 P 4757283204f046669da4e6f478fa5369: Time spent starting tablet: real 0.016s user 0.008s sys 0.009s
I20260928 03:04:12.474583 7945 heartbeater.cc:503] Master 127.6.192.62:40141 was elected leader, sending a full tablet report...
I20260928 03:04:12.475342 7964 ts_tablet_manager.cc:1428] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
I20260928 03:04:12.512029 7971 raft_consensus.cc:482] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:04:12.512414 7971 raft_consensus.cc:504] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4757283204f046669da4e6f478fa5369" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41693 } } peers { permanent_uuid: "b8166bb0ebc64efb8f92f7a41b3e40f7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44153 } } peers { permanent_uuid: "bc6b842a983a4805968921ef013e3a7b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 40415 } }
I20260928 03:04:12.514427 7971 leader_election.cc:288] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4757283204f046669da4e6f478fa5369 (127.6.192.2:41693), bc6b842a983a4805968921ef013e3a7b (127.6.192.1:40415)
W20260928 03:04:12.519289 7678 tablet.cc:2369] T e5be4c5f218e48b78351dfdec03abfb6 P bc6b842a983a4805968921ef013e3a7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:04:12.522027 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.514836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:41693 (local address 127.6.192.3:33789)
0928 03:04:12.515095 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:12.515114 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:12.515302 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:12.516120 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:12.516126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:12.516159 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:12.516384 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:12.516397 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.517690 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.517696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:12.519610 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:12.519632 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.521429 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.521434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:12.521583 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:12.521608 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 03:04:12.521689 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 03:04:12.521794 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":3}
I20260928 03:04:12.522221 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.515203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:33789 (local address 127.6.192.2:41693)
0928 03:04:12.515426 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:12.515434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:12.515487 (+ 53us) server_negotiation.cc:407] Connection header received
0928 03:04:12.515701 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:12.515707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:12.515802 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:12.515900 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:12.516563 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.517528 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.519875 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.521228 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.521406 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:12.521521 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:12.521756 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 03:04:12.521917 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 03:04:12.523085 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.515274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:40415 (local address 127.6.192.3:56155)
0928 03:04:12.515978 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:12.515993 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:12.516101 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:12.516411 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:12.516414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:12.516432 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:12.516937 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:12.516952 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.518480 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.518484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:12.520567 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:12.520583 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.522495 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.522500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:12.522630 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:12.522653 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 03:04:12.522728 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 03:04:12.522822 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":53,"thread_start_us":417,"threads_started":1}
I20260928 03:04:12.523270 7766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5be4c5f218e48b78351dfdec03abfb6" candidate_uuid: "b8166bb0ebc64efb8f92f7a41b3e40f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4757283204f046669da4e6f478fa5369" is_pre_election: true
I20260928 03:04:12.523763 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.515635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:56155 (local address 127.6.192.1:40415)
0928 03:04:12.515890 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:12.515897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:12.516085 (+ 188us) server_negotiation.cc:407] Connection header received
0928 03:04:12.516200 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:12.516206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:12.516284 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:12.516391 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:12.517135 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.518298 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.520758 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.522186 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.522356 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:12.522476 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:12.523324 (+ 848us) server_negotiation.cc:299] Negotiation successful
0928 03:04:12.523550 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 03:04:12.524015 7766 raft_consensus.cc:2454] T e5be4c5f218e48b78351dfdec03abfb6 P 4757283204f046669da4e6f478fa5369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8166bb0ebc64efb8f92f7a41b3e40f7 in term 0.
I20260928 03:04:12.524856 7835 leader_election.cc:302] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4757283204f046669da4e6f478fa5369, b8166bb0ebc64efb8f92f7a41b3e40f7; no voters:
I20260928 03:04:12.524807 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5be4c5f218e48b78351dfdec03abfb6" candidate_uuid: "b8166bb0ebc64efb8f92f7a41b3e40f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc6b842a983a4805968921ef013e3a7b" is_pre_election: true
I20260928 03:04:12.525290 7971 raft_consensus.cc:2781] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:04:12.525405 7971 raft_consensus.cc:482] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:04:12.525400 7632 raft_consensus.cc:2454] T e5be4c5f218e48b78351dfdec03abfb6 P bc6b842a983a4805968921ef013e3a7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8166bb0ebc64efb8f92f7a41b3e40f7 in term 0.
I20260928 03:04:12.525535 7971 raft_consensus.cc:3037] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:12.527987 7971 raft_consensus.cc:504] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4757283204f046669da4e6f478fa5369" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41693 } } peers { permanent_uuid: "b8166bb0ebc64efb8f92f7a41b3e40f7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44153 } } peers { permanent_uuid: "bc6b842a983a4805968921ef013e3a7b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 40415 } }
I20260928 03:04:12.528589 7971 leader_election.cc:288] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7 [CANDIDATE]: Term 1 election: Requested vote from peers 4757283204f046669da4e6f478fa5369 (127.6.192.2:41693), bc6b842a983a4805968921ef013e3a7b (127.6.192.1:40415)
I20260928 03:04:12.529215 7766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5be4c5f218e48b78351dfdec03abfb6" candidate_uuid: "b8166bb0ebc64efb8f92f7a41b3e40f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4757283204f046669da4e6f478fa5369"
I20260928 03:04:12.529439 7766 raft_consensus.cc:3037] T e5be4c5f218e48b78351dfdec03abfb6 P 4757283204f046669da4e6f478fa5369 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:12.529291 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5be4c5f218e48b78351dfdec03abfb6" candidate_uuid: "b8166bb0ebc64efb8f92f7a41b3e40f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc6b842a983a4805968921ef013e3a7b"
I20260928 03:04:12.529647 7632 raft_consensus.cc:3037] T e5be4c5f218e48b78351dfdec03abfb6 P bc6b842a983a4805968921ef013e3a7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:12.532168 7766 raft_consensus.cc:2454] T e5be4c5f218e48b78351dfdec03abfb6 P 4757283204f046669da4e6f478fa5369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8166bb0ebc64efb8f92f7a41b3e40f7 in term 1.
I20260928 03:04:12.532763 7632 raft_consensus.cc:2454] T e5be4c5f218e48b78351dfdec03abfb6 P bc6b842a983a4805968921ef013e3a7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8166bb0ebc64efb8f92f7a41b3e40f7 in term 1.
I20260928 03:04:12.532801 7835 leader_election.cc:302] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4757283204f046669da4e6f478fa5369, b8166bb0ebc64efb8f92f7a41b3e40f7; no voters:
I20260928 03:04:12.533185 7971 raft_consensus.cc:2781] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:12.533828 7971 raft_consensus.cc:686] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7 [term 1 LEADER]: Becoming Leader. State: Replica: b8166bb0ebc64efb8f92f7a41b3e40f7, State: Running, Role: LEADER
I20260928 03:04:12.534368 7971 consensus_queue.cc:237] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4757283204f046669da4e6f478fa5369" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41693 } } peers { permanent_uuid: "b8166bb0ebc64efb8f92f7a41b3e40f7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44153 } } peers { permanent_uuid: "bc6b842a983a4805968921ef013e3a7b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 40415 } }
I20260928 03:04:12.539434 7488 catalog_manager.cc:5495] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8166bb0ebc64efb8f92f7a41b3e40f7 (127.6.192.3). New cstate: current_term: 1 leader_uuid: "b8166bb0ebc64efb8f92f7a41b3e40f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4757283204f046669da4e6f478fa5369" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8166bb0ebc64efb8f92f7a41b3e40f7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc6b842a983a4805968921ef013e3a7b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 40415 } health_report { overall_health: UNKNOWN } } }
W20260928 03:04:12.548247 7812 tablet.cc:2369] T e5be4c5f218e48b78351dfdec03abfb6 P 4757283204f046669da4e6f478fa5369: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 03:04:12 dist-test-slave-fjww krb5kdc[7443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:04:12.563735 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.547686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34520 (local address 127.6.192.1:40415)
0928 03:04:12.547933 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:12.547941 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:12.548378 (+ 437us) server_negotiation.cc:407] Connection header received
0928 03:04:12.548640 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:12.548648 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:12.548768 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:12.548886 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:12.550349 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.551129 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.552548 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.553048 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.553147 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:12.557389 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:12.557405 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:12.557467 (+ 62us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:12.557541 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:12.560692 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:12.561574 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:12.561581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:12.561583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:12.561714 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:12.562123 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:12.562126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:12.562128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:12.562497 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:12.562723 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:12.562989 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 03:04:12.563211 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 03:04:12.578965 7946 tablet.cc:2369] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 03:04:12 dist-test-slave-fjww krb5kdc[7443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564652, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 03:04:12 dist-test-slave-fjww krb5kdc[7443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564652, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:04:12.612483 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.597187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.6.192.1:40415)
0928 03:04:12.597383 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:12.597390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:12.597855 (+ 465us) server_negotiation.cc:407] Connection header received
0928 03:04:12.598061 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:12.598067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:12.598145 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:12.598257 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:12.599872 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.600629 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.601992 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.602426 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.602580 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:12.606846 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:12.606851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:12.606854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:12.606897 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:12.609669 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:12.610377 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:12.610383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:12.610389 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:12.610547 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:12.610987 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:12.610995 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:12.610998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:12.611290 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:12.611602 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:12.611894 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 03:04:12.612132 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 03:04:12 dist-test-slave-fjww krb5kdc[7443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 03:04:12 dist-test-slave-fjww krb5kdc[7443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:04:12.665618 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.646100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34528 (local address 127.6.192.1:40415)
0928 03:04:12.646388 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:12.646394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:12.646690 (+ 296us) server_negotiation.cc:407] Connection header received
0928 03:04:12.646830 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:12.646834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:12.646910 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:12.647002 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:12.648652 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.649391 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.650598 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.651048 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.651146 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:12.655235 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:12.655240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:12.655244 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:12.655286 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:12.662529 (+ 7243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:12.663401 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:12.663406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:12.663409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:12.663691 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:12.664134 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:12.664138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:12.664140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:12.664488 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:12.664689 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:12.665047 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 03:04:12.665249 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260928 03:04:12.667681 7592 tablet_service.cc:2908] Rejecting scan request for tablet e5be4c5f218e48b78351dfdec03abfb6: Uninitialized: clean time has not yet been initialized
I20260928 03:04:12.671929 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 7549
I20260928 03:04:12.686115 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.672898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.6.192.1:40415)
0928 03:04:12.673127 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:12.673133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:12.673411 (+ 278us) server_negotiation.cc:407] Connection header received
0928 03:04:12.673550 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:12.673553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:12.673629 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:12.673714 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:12.675203 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.676097 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.677529 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.677967 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.678080 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:12.680741 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:12.680746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:12.680749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:12.680784 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:12.683598 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:12.684305 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:12.684309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:12.684311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:12.684443 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:12.684792 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:12.684795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:12.684797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:12.685084 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:12.685294 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:12.685567 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 03:04:12.685740 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 03:04:12.686960 7672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:12.778470 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 7549
I20260928 03:04:12.796821 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 7683
Sep 28 03:04:12 dist-test-slave-fjww krb5kdc[7443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.2@KRBTEST.COM
I20260928 03:04:12.816478 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.798084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.6.192.2:41693)
0928 03:04:12.798306 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:12.798312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:12.798332 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:04:12.798405 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:12.798409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:12.798522 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:12.798631 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:12.800667 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.801868 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.803872 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.804546 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.804643 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:12.809885 (+ 5242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:12.809898 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:12.809916 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:12.809968 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:12.813601 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:12.814392 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:12.814403 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:12.814406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:12.814542 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:12.814928 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:12.814932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:12.814935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:12.815356 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:12.815808 (+ 452us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:12.816034 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 03:04:12.816221 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 03:04:12.817204 7806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:12.924484 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 7683
I20260928 03:04:12.943125 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 7817
I20260928 03:04:12.950101 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.949664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:41693 (local address 127.6.192.3:35649)
0928 03:04:12.949834 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:12.949962 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.2:41693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 03:04:12.950424 7835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.192.2:41693: connect: Connection refused (error 111)
I20260928 03:04:12.952150 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.951636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:41693 (local address 127.6.192.3:56815)
0928 03:04:12.951866 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:12.951967 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.2:41693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
W20260928 03:04:12.952373 7835 consensus_peers.cc:487] T e5be4c5f218e48b78351dfdec03abfb6 P b8166bb0ebc64efb8f92f7a41b3e40f7 -> Peer 4757283204f046669da4e6f478fa5369 (127.6.192.2:41693): Couldn't send request to peer 4757283204f046669da4e6f478fa5369. Status: Network error: Client connection negotiation failed: client connection to 127.6.192.2:41693: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Sep 28 03:04:12 dist-test-slave-fjww krb5kdc[7443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.3@KRBTEST.COM
I20260928 03:04:12.959385 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:12.944020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44550 (local address 127.6.192.3:44153)
0928 03:04:12.944479 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:12.944484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:12.944500 (+ 16us) server_negotiation.cc:407] Connection header received
0928 03:04:12.944557 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:12.944561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:12.944621 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:12.944721 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:12.946013 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.946758 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.948136 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:12.948677 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:12.948817 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:12.953629 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:12.953640 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:12.953655 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:12.953694 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:12.956662 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:12.957386 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:12.957402 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:12.957406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:12.957532 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:12.957963 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:12.957969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:12.957972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:12.958353 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:12.958655 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:12.958928 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 03:04:12.959113 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":255,"threads_started":1}
I20260928 03:04:12.960107 7940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:13.052357 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 7817
I20260928 03:04:13.069229 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 7456
I20260928 03:04:13.070436 7518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:13.178951 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 7456
2026-09-28T03:04:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4320 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:04:13 dist-test-slave-fjww krb5kdc[7995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:04:13 dist-test-slave-fjww krb5kdc[7995](info): set up 2 sockets
Sep 28 03:04:13 dist-test-slave-fjww krb5kdc[7995](info): commencing operation
krb5kdc: starting...
W20260928 03:04:14.271952 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:04:14 dist-test-slave-fjww krb5kdc[7995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:04:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:04:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:04:14.473318 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:35331
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:35331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:14.812268 8008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:14.812637 8008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:14.812809 8008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:14.820521 8008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:04:14.820690 8008 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 03:04:14.820785 8008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:14.820905 8008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:14.820988 8008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:04:14.821075 8008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:04:14.831611 8008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:35331
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:35331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:14.833006 8008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:14.834409 8008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:14.847276 8013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:14.847155 8016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:14.847108 8014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:14.848978 8008 server_base.cc:1034] running on GCE node
I20260928 03:04:14.849975 8008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:14.851749 8008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:14.853057 8008 hybrid_clock.cc:648] HybridClock initialized: now 1790564654852963 us; error 69 us; skew 500 ppm
I20260928 03:04:14.853542 8008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:14 dist-test-slave-fjww krb5kdc[7995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564654, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:14.857131 8008 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:04:14.858932 8008 webserver.cc:460] Webserver started at http://127.6.192.62:37597/ using document root <none> and password file <none>
I20260928 03:04:14.859789 8008 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:14.859930 8008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:14.860263 8008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:14.863505 8008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "2d3fa10d7bea40d49b73dd5944aec000"
format_stamp: "Formatted at 2026-09-28 03:04:14 on dist-test-slave-fjww"
server_key: "39c9c1d2b9b1e449ff6b1cdc42af44d1"
server_key_iv: "9d71d180c3e30eed4f709626c687da1b"
server_key_version: "encryptionkey@0"
I20260928 03:04:14.864382 8008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "2d3fa10d7bea40d49b73dd5944aec000"
format_stamp: "Formatted at 2026-09-28 03:04:14 on dist-test-slave-fjww"
server_key: "39c9c1d2b9b1e449ff6b1cdc42af44d1"
server_key_iv: "9d71d180c3e30eed4f709626c687da1b"
server_key_version: "encryptionkey@0"
I20260928 03:04:14.869895 8008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:04:14.873925 8023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:14.875720 8008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:04:14.875931 8008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "2d3fa10d7bea40d49b73dd5944aec000"
format_stamp: "Formatted at 2026-09-28 03:04:14 on dist-test-slave-fjww"
server_key: "39c9c1d2b9b1e449ff6b1cdc42af44d1"
server_key_iv: "9d71d180c3e30eed4f709626c687da1b"
server_key_version: "encryptionkey@0"
I20260928 03:04:14.876170 8008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:14.893028 8008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:14.897778 8008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:14.898077 8008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:14.921654 8008 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:35331
I20260928 03:04:14.921674 8084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:35331 every 8 connection(s)
I20260928 03:04:14.923197 8008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:04:14.923745 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 8008
I20260928 03:04:14.924017 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:04:14.924925 6912 external_mini_cluster.cc:1428] Setting key 13e3ebf8939bce63d54136f668856efb
I20260928 03:04:14.928471 8085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 03:04:14 dist-test-slave-fjww krb5kdc[7995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:14.942862 8085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000: Bootstrap starting.
I20260928 03:04:14.946218 8085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:14.947721 8085 log.cc:824] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:14.950724 8085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000: No bootstrap required, opened a new log
I20260928 03:04:14.952556 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:14.927606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58548 (local address 127.6.192.62:35331)
0928 03:04:14.928616 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:14.928633 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:14.928692 (+ 59us) server_negotiation.cc:407] Connection header received
0928 03:04:14.930393 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:14.930417 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:14.930851 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:14.931254 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:14.933425 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:14.935051 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:14.936969 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:14.937840 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:14.937995 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:14.943261 (+ 5266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:14.943362 (+ 101us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:14.943383 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:14.943435 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:14.947341 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:14.948430 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:14.948482 (+ 52us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:14.948494 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:14.948831 (+ 337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:14.949295 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:14.949301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:14.949305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:14.949921 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:14.950250 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:14.950764 (+ 514us) server_negotiation.cc:299] Negotiation successful
0928 03:04:14.951315 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":475,"threads_started":1}
I20260928 03:04:14.958601 8085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3fa10d7bea40d49b73dd5944aec000" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 35331 } }
I20260928 03:04:14.959030 8085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:14.959151 8085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3fa10d7bea40d49b73dd5944aec000, State: Initialized, Role: FOLLOWER
I20260928 03:04:14.959818 8085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3fa10d7bea40d49b73dd5944aec000" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 35331 } }
I20260928 03:04:14.960041 8085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:04:14.960194 8085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:04:14.960347 8085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:14.962534 8085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3fa10d7bea40d49b73dd5944aec000" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 35331 } }
I20260928 03:04:14.963099 8085 leader_election.cc:302] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d3fa10d7bea40d49b73dd5944aec000; no voters:
I20260928 03:04:14.963806 8085 leader_election.cc:288] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:04:14.963991 8090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:14.964864 8090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000 [term 1 LEADER]: Becoming Leader. State: Replica: 2d3fa10d7bea40d49b73dd5944aec000, State: Running, Role: LEADER
I20260928 03:04:14.965560 8090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3fa10d7bea40d49b73dd5944aec000" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 35331 } }
I20260928 03:04:14.966137 8085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:04:14.969933 8091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d3fa10d7bea40d49b73dd5944aec000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3fa10d7bea40d49b73dd5944aec000" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 35331 } } }
I20260928 03:04:14.970399 8091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:14.970070 8092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d3fa10d7bea40d49b73dd5944aec000. Latest consensus state: current_term: 1 leader_uuid: "2d3fa10d7bea40d49b73dd5944aec000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3fa10d7bea40d49b73dd5944aec000" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 35331 } } }
I20260928 03:04:14.970621 8092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:14.971493 8096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:04:14.977862 8096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:04:14.991623 8096 catalog_manager.cc:1347] Generated new cluster ID: 95f4d577c32340a4811add6823f03619
I20260928 03:04:14.991780 8096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:04:15.018900 8096 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:04:15.021211 8096 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:04:15.032759 8096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000: Generated new TSK 0
I20260928 03:04:15.033859 8096 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
I20260928 03:04:15.096599 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:35331
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:15.445991 8111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:15.446393 8111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:15.446524 8111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:15.454164 8111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:15.454375 8111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:15.454591 8111 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 03:04:15.454701 8111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:04:15.465349 8111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.192.62:35331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:15.466766 8111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:15.468272 8111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:15.482228 8117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:15.482453 8116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:15.482482 8119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:15.483723 8111 server_base.cc:1034] running on GCE node
I20260928 03:04:15.484450 8111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:15.485503 8111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:15.486773 8111 hybrid_clock.cc:648] HybridClock initialized: now 1790564655486680 us; error 65 us; skew 500 ppm
I20260928 03:04:15.487108 8111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:15 dist-test-slave-fjww krb5kdc[7995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564655, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:15.490782 8111 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:04:15.492419 8111 webserver.cc:460] Webserver started at http://127.6.192.1:42613/ using document root <none> and password file <none>
I20260928 03:04:15.493157 8111 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:15.493294 8111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:15.493642 8111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:15.496891 8111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0"
format_stamp: "Formatted at 2026-09-28 03:04:15 on dist-test-slave-fjww"
server_key: "0299243da7461baf2d41c6b9d4455018"
server_key_iv: "4cb8ea12d8ce6fb702e9deb129dfe13b"
server_key_version: "encryptionkey@0"
I20260928 03:04:15.497774 8111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0"
format_stamp: "Formatted at 2026-09-28 03:04:15 on dist-test-slave-fjww"
server_key: "0299243da7461baf2d41c6b9d4455018"
server_key_iv: "4cb8ea12d8ce6fb702e9deb129dfe13b"
server_key_version: "encryptionkey@0"
I20260928 03:04:15.503841 8111 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 03:04:15.507838 8126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:15.509322 8111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:04:15.509574 8111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0"
format_stamp: "Formatted at 2026-09-28 03:04:15 on dist-test-slave-fjww"
server_key: "0299243da7461baf2d41c6b9d4455018"
server_key_iv: "4cb8ea12d8ce6fb702e9deb129dfe13b"
server_key_version: "encryptionkey@0"
I20260928 03:04:15.509821 8111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:15.523012 8111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:15.527421 8111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:15.527829 8111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:15.530754 8111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:15.530869 8111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:15.531064 8111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:15.531150 8111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 03:04:15 dist-test-slave-fjww krb5kdc[7995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564655, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:15.556094 8135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:15.531353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:33781 (local address 127.6.192.62:35331)
0928 03:04:15.532198 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:15.532203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:15.533381 (+ 1178us) server_negotiation.cc:407] Connection header received
0928 03:04:15.535761 (+ 2380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:15.535767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:15.535870 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:15.535993 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:15.538320 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:15.539419 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:15.542179 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:15.542584 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:15.542724 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:15.548140 (+ 5416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:15.548148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:15.548153 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:15.548203 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:15.552327 (+ 4124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:15.553547 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:15.553554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:15.553558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:15.553703 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:15.554213 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:15.554219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:15.554224 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:15.554520 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:15.554681 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:15.555472 (+ 791us) server_negotiation.cc:299] Negotiation successful
0928 03:04:15.555707 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":643,"threads_started":1}
I20260928 03:04:15.557240 8136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:15.532138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:35331 (local address 127.6.192.1:33781)
0928 03:04:15.533194 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:15.533237 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:15.535374 (+ 2137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:15.536266 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:15.536276 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:15.537106 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:15.538128 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:15.538150 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:15.539642 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:15.539646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:15.541932 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:15.541957 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:15.542889 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:15.542903 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:15.543290 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:15.544236 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:15.544280 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:15.547800 (+ 3520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:15.552543 (+ 4743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:15.552558 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:15.552585 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:15.553390 (+ 805us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:15.553832 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:15.553838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:15.553842 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:15.554070 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:15.554657 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:15.554666 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:15.555060 (+ 394us) client_negotiation.cc:769] Sending connection context
0928 03:04:15.555359 (+ 299us) client_negotiation.cc:240] Negotiation successful
0928 03:04:15.556002 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":510,"threads_started":1}
I20260928 03:04:15.583171 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:15.564981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:35331 (local address 127.6.192.1:46691)
0928 03:04:15.565958 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:15.565985 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:15.566186 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:15.566568 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:15.566576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:15.567408 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:15.568080 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:15.568103 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:15.569654 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:15.569663 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:15.572121 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:15.572146 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:15.573006 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:15.573019 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:15.573219 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:15.574111 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:15.574143 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:15.575385 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:15.580222 (+ 4837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:15.580229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:15.580234 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:15.580717 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:15.581305 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:15.581311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:15.581315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:15.581462 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:15.582217 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:15.582221 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:15.582412 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 03:04:15.582521 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 03:04:15.582819 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":467,"threads_started":1}
I20260928 03:04:15.583570 8135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:15.564942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:46691 (local address 127.6.192.62:35331)
0928 03:04:15.565178 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:15.565187 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:15.566109 (+ 922us) server_negotiation.cc:407] Connection header received
0928 03:04:15.566344 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:15.566348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:15.566417 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:15.566494 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:15.568361 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:15.569495 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:15.572328 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:15.572775 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:15.572887 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:15.575666 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:15.575677 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:15.575685 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:15.575741 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:15.580038 (+ 4297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:15.580882 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:15.580891 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:15.580906 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:15.581115 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:15.581587 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:15.581595 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:15.581601 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:15.582080 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:15.582261 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:15.582693 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 03:04:15.583026 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 03:04:15.588220 8111 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:39393
I20260928 03:04:15.588243 8246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:39393 every 8 connection(s)
I20260928 03:04:15.589493 8111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:04:15.591856 8247 heartbeater.cc:347] Connected to a master server at 127.6.192.62:35331
I20260928 03:04:15.592113 8247 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:15.592818 8247 heartbeater.cc:511] Master 127.6.192.62:35331 requested a full tablet report, sending...
I20260928 03:04:15.594961 8040 ts_manager.cc:194] Registered new tserver with Master: 3ee9eb7bfd6d4567b10145bb7034cbc0 (127.6.192.1:39393)
I20260928 03:04:15.596498 8040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:33781
I20260928 03:04:15.599622 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 8111
I20260928 03:04:15.599861 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:04:15.600528 6912 external_mini_cluster.cc:1428] Setting key 28b30e178d6c3185076bec93fe6f7a32
WARNING: no policy specified for kudu/127.6.192.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
I20260928 03:04:15.647107 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:35331
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:15.993605 8251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:15.993934 8251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:15.994045 8251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:16.002295 8251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:16.002439 8251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:16.002527 8251 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 03:04:16.002609 8251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:04:16.012923 8251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.192.62:35331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:16.014367 8251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:16.015918 8251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:16.030169 8256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:16.030104 8257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:16.030054 8259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:16.031338 8251 server_base.cc:1034] running on GCE node
I20260928 03:04:16.032349 8251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:16.033445 8251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:16.034763 8251 hybrid_clock.cc:648] HybridClock initialized: now 1790564656034683 us; error 51 us; skew 500 ppm
I20260928 03:04:16.035141 8251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:16 dist-test-slave-fjww krb5kdc[7995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564656, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:16.038789 8251 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:04:16.040540 8251 webserver.cc:460] Webserver started at http://127.6.192.2:34431/ using document root <none> and password file <none>
I20260928 03:04:16.041363 8251 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:16.041517 8251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:16.041862 8251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:16.045087 8251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "5f83e90b75d546d08d108940f6c0347b"
format_stamp: "Formatted at 2026-09-28 03:04:16 on dist-test-slave-fjww"
server_key: "4c3151d6b9fc33f6d0dd75ec4b07a856"
server_key_iv: "c41260a3171cb56b15d33e32be785411"
server_key_version: "encryptionkey@0"
I20260928 03:04:16.046120 8251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "5f83e90b75d546d08d108940f6c0347b"
format_stamp: "Formatted at 2026-09-28 03:04:16 on dist-test-slave-fjww"
server_key: "4c3151d6b9fc33f6d0dd75ec4b07a856"
server_key_iv: "c41260a3171cb56b15d33e32be785411"
server_key_version: "encryptionkey@0"
I20260928 03:04:16.053102 8251 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 03:04:16.057408 8266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:16.059242 8251 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 03:04:16.059456 8251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "5f83e90b75d546d08d108940f6c0347b"
format_stamp: "Formatted at 2026-09-28 03:04:16 on dist-test-slave-fjww"
server_key: "4c3151d6b9fc33f6d0dd75ec4b07a856"
server_key_iv: "c41260a3171cb56b15d33e32be785411"
server_key_version: "encryptionkey@0"
I20260928 03:04:16.059814 8251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:16.088029 8251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:16.093716 8251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:16.094012 8251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:16.097388 8251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:16.097543 8251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:16.097745 8251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:16.097855 8251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 03:04:16 dist-test-slave-fjww krb5kdc[7995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564656, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:16.125208 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:16.097642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:38023 (local address 127.6.192.62:35331)
0928 03:04:16.098369 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:16.098377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:16.099691 (+ 1314us) server_negotiation.cc:407] Connection header received
0928 03:04:16.101897 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:16.101909 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:16.102033 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:16.102197 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:16.104542 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.106194 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.109103 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.109656 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.109819 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:16.117303 (+ 7484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:16.117309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:16.117313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:16.117351 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:16.120635 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:16.121860 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:16.121865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:16.121867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:16.122008 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:16.122612 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:16.122623 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:16.122625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:16.122979 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:16.123179 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:16.124709 (+ 1530us) server_negotiation.cc:299] Negotiation successful
0928 03:04:16.124929 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":382,"threads_started":1}
I20260928 03:04:16.127991 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:16.098506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:35331 (local address 127.6.192.2:38023)
0928 03:04:16.099482 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:16.099530 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:16.101491 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:16.102464 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:16.102474 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:16.103087 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:16.104245 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:16.104285 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.106451 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.106456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:16.108873 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:16.108894 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.110006 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.110025 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:16.110409 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:16.111513 (+ 1104us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:16.111572 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:16.116927 (+ 5355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:16.120886 (+ 3959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:16.120900 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:16.120940 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:16.121628 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:16.122137 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:16.122142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:16.122146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:16.122469 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:16.123177 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:16.123195 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:16.124049 (+ 854us) client_negotiation.cc:769] Sending connection context
0928 03:04:16.124574 (+ 525us) client_negotiation.cc:240] Negotiation successful
0928 03:04:16.125491 (+ 917us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":399,"threads_started":1}
I20260928 03:04:16.153230 8251 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:41729
I20260928 03:04:16.153179 8386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:41729 every 8 connection(s)
I20260928 03:04:16.155027 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:16.137525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:41733 (local address 127.6.192.62:35331)
0928 03:04:16.137724 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:16.137732 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:16.138412 (+ 680us) server_negotiation.cc:407] Connection header received
0928 03:04:16.138705 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:16.138710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:16.138793 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:16.138901 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:16.140092 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.141117 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.143198 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.143811 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.143962 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:16.147180 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:16.147185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:16.147188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:16.147221 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:16.151488 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:16.152476 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:16.152488 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:16.152492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:16.152734 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:16.153440 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:16.153445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:16.153449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:16.153814 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:16.153933 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:16.154567 (+ 634us) server_negotiation.cc:299] Negotiation successful
0928 03:04:16.154790 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 03:04:16.155098 8251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:04:16.155753 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:16.137542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:35331 (local address 127.6.192.2:41733)
0928 03:04:16.138270 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:16.138298 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:16.138564 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:16.138938 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:16.138943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:16.139549 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:16.139936 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:16.139947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.141299 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.141303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:16.142964 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:16.142978 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.144082 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.144096 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:16.144402 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:16.145272 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:16.145315 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:16.146935 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:16.151715 (+ 4780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:16.151723 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:16.151730 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:16.152342 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:16.152952 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:16.152962 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:16.152970 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:16.153192 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:16.153958 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:16.153970 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:16.154228 (+ 258us) client_negotiation.cc:769] Sending connection context
0928 03:04:16.154436 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 03:04:16.154876 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":425,"threads_started":1}
I20260928 03:04:16.158123 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 8251
I20260928 03:04:16.158368 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:04:16.159096 6912 external_mini_cluster.cc:1428] Setting key 661b7bfc93d619dcfaf75fc6612d827c
I20260928 03:04:16.160197 8387 heartbeater.cc:347] Connected to a master server at 127.6.192.62:35331
I20260928 03:04:16.160593 8387 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:16.161568 8387 heartbeater.cc:511] Master 127.6.192.62:35331 requested a full tablet report, sending...
I20260928 03:04:16.163141 8040 ts_manager.cc:194] Registered new tserver with Master: 5f83e90b75d546d08d108940f6c0347b (127.6.192.2:41729)
I20260928 03:04:16.164600 8040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:38023
WARNING: no policy specified for kudu/127.6.192.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
I20260928 03:04:16.204064 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:35331
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:16.559994 8391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:16.560513 8391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:16.560683 8391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:16.568406 8391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:16.568581 8391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:16.568740 8391 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 03:04:16.568892 8391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:04:16.578763 8391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.192.62:35331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:16.580317 8391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:16.581868 8391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:16.597121 8396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:16.597220 8397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:16.597244 8399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:16.598325 8391 server_base.cc:1034] running on GCE node
I20260928 03:04:16.598989 8391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:16.599345 8247 heartbeater.cc:503] Master 127.6.192.62:35331 was elected leader, sending a full tablet report...
I20260928 03:04:16.600181 8391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:16.601432 8391 hybrid_clock.cc:648] HybridClock initialized: now 1790564656601354 us; error 53 us; skew 500 ppm
I20260928 03:04:16.601714 8391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:16 dist-test-slave-fjww krb5kdc[7995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564656, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:16.605123 8391 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:04:16.606834 8391 webserver.cc:460] Webserver started at http://127.6.192.3:33185/ using document root <none> and password file <none>
I20260928 03:04:16.607654 8391 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:16.607750 8391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:16.608108 8391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:16.611184 8391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "f2ed1a82633e406abcbc0fe46390f800"
format_stamp: "Formatted at 2026-09-28 03:04:16 on dist-test-slave-fjww"
server_key: "c18ecbe4e0495f4891c10af46a741bda"
server_key_iv: "d28fa1da0e0e9b200b872d534d3e7e43"
server_key_version: "encryptionkey@0"
I20260928 03:04:16.612145 8391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "f2ed1a82633e406abcbc0fe46390f800"
format_stamp: "Formatted at 2026-09-28 03:04:16 on dist-test-slave-fjww"
server_key: "c18ecbe4e0495f4891c10af46a741bda"
server_key_iv: "d28fa1da0e0e9b200b872d534d3e7e43"
server_key_version: "encryptionkey@0"
I20260928 03:04:16.617727 8391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:04:16.621609 8406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:16.623265 8391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 03:04:16.623507 8391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "f2ed1a82633e406abcbc0fe46390f800"
format_stamp: "Formatted at 2026-09-28 03:04:16 on dist-test-slave-fjww"
server_key: "c18ecbe4e0495f4891c10af46a741bda"
server_key_iv: "d28fa1da0e0e9b200b872d534d3e7e43"
server_key_version: "encryptionkey@0"
I20260928 03:04:16.623749 8391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:16.636934 8391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:16.641481 8391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:16.641790 8391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:16.644796 8391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:16.644927 8391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:16.645115 8391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:16.645190 8391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 03:04:16 dist-test-slave-fjww krb5kdc[7995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564656, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:16.673314 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:16.645288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:44507 (local address 127.6.192.62:35331)
0928 03:04:16.645551 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:16.645561 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:16.647163 (+ 1602us) server_negotiation.cc:407] Connection header received
0928 03:04:16.649505 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:16.649510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:16.649609 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:16.649715 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:16.652746 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.653969 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.657433 (+ 3464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.658113 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.658272 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:16.663768 (+ 5496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:16.663795 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:16.663802 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:16.663858 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:16.668233 (+ 4375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:16.669596 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:16.669606 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:16.669612 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:16.669818 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:16.670509 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:16.670518 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:16.670525 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:16.671011 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:16.671188 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:16.672468 (+ 1280us) server_negotiation.cc:299] Negotiation successful
0928 03:04:16.672787 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 03:04:16.675097 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:16.646038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:35331 (local address 127.6.192.3:44507)
0928 03:04:16.646972 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:16.647017 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:16.649173 (+ 2156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:16.650067 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:16.650085 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:16.651164 (+ 1079us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:16.652413 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:16.652453 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.654200 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.654206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:16.657224 (+ 3018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:16.657248 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.658456 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.658466 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:16.658862 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:16.659997 (+ 1135us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:16.660039 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:16.663347 (+ 3308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:16.668454 (+ 5107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:16.668468 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:16.668496 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:16.669385 (+ 889us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:16.669993 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:16.670000 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:16.670006 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:16.670340 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:16.671215 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:16.671224 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:16.671889 (+ 665us) client_negotiation.cc:769] Sending connection context
0928 03:04:16.672306 (+ 417us) client_negotiation.cc:240] Negotiation successful
0928 03:04:16.673076 (+ 770us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":406,"threads_started":1}
I20260928 03:04:16.687649 8391 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:35031
I20260928 03:04:16.687672 8525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:35031 every 8 connection(s)
I20260928 03:04:16.689200 8391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:04:16.691922 8526 heartbeater.cc:347] Connected to a master server at 127.6.192.62:35331
I20260928 03:04:16.692199 8526 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:16.693042 8526 heartbeater.cc:511] Master 127.6.192.62:35331 requested a full tablet report, sending...
I20260928 03:04:16.694775 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 8391
I20260928 03:04:16.694996 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:04:16.695003 8040 ts_manager.cc:194] Registered new tserver with Master: f2ed1a82633e406abcbc0fe46390f800 (127.6.192.3:35031)
I20260928 03:04:16.695487 6912 external_mini_cluster.cc:1428] Setting key eba4e1ceca637562bbeb20de405e31f0
I20260928 03:04:16.696256 8040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:44507
I20260928 03:04:16.699270 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:16.684236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:35331 (local address 127.6.192.3:39713)
0928 03:04:16.684916 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:16.684945 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:16.685150 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:16.685536 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:16.685540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:16.686021 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:16.686366 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:16.686377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.687672 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.687677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:16.688767 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:16.688788 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.689693 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.689702 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:16.689914 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:16.690741 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:16.690779 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:16.691845 (+ 1066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:16.695868 (+ 4023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:16.695874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:16.695878 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:16.696856 (+ 978us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:16.697317 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:16.697345 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:16.697349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:16.697515 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:16.698082 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:16.698088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:16.698279 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 03:04:16.698437 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 03:04:16.698822 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":354,"threads_started":1}
I20260928 03:04:16.699489 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:04:16.699527 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:16.684565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:39713 (local address 127.6.192.62:35331)
0928 03:04:16.684804 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:16.684814 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:16.685061 (+ 247us) server_negotiation.cc:407] Connection header received
0928 03:04:16.685310 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:16.685315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:16.685380 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:16.685499 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:16.686533 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.687417 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.688944 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.689465 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.689616 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:16.692047 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:16.692055 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:16.692061 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:16.692108 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:16.695707 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:16.696991 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:16.696996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:16.696999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:16.697174 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:16.697636 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:16.697644 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:16.697647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:16.697950 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:16.698106 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:16.698545 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 03:04:16.698804 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 03:04:16 dist-test-slave-fjww krb5kdc[7995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564656, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 03:04:16 dist-test-slave-fjww krb5kdc[7995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564656, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:16.748601 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:16.733968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58554 (local address 127.6.192.62:35331)
0928 03:04:16.734151 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:16.734156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:16.734574 (+ 418us) server_negotiation.cc:407] Connection header received
0928 03:04:16.734735 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:16.734739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:16.734802 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:16.734896 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:16.736469 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.737217 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.738523 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.738905 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.739011 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:16.743208 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:16.743213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:16.743217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:16.743258 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:16.746010 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:16.746687 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:16.746692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:16.746694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:16.746846 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:16.747306 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:16.747311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:16.747313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:16.747673 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:16.747863 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:16.748184 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 03:04:16.748364 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 03:04:16.752760 8040 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:04:16.760177 8040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:04:16.786660 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:16.774525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.6.192.1:39393)
0928 03:04:16.775367 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:16.775380 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:16.776984 (+ 1604us) server_negotiation.cc:407] Connection header received
0928 03:04:16.777303 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:16.777313 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:16.777586 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:16.777756 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:16.778756 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.780429 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.783399 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.785025 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.785128 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:16.785390 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:16.786132 (+ 742us) server_negotiation.cc:299] Negotiation successful
0928 03:04:16.786321 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":420,"threads_started":1}
I20260928 03:04:16.787498 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:16.773875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:39393 (local address 127.0.0.1:33336)
0928 03:04:16.776854 (+ 2979us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:16.776871 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:16.777044 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:16.778109 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:16.778117 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:16.778155 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:16.778570 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:16.778583 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.780659 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.780664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:16.783195 (+ 2531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:16.783210 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.785595 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.785601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:16.785856 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:16.785918 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 03:04:16.786907 (+ 989us) client_negotiation.cc:240] Negotiation successful
0928 03:04:16.787112 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2853,"thread_start_us":2262,"threads_started":1}
I20260928 03:04:16.789464 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:16.774007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55924 (local address 127.6.192.3:35031)
0928 03:04:16.775953 (+ 1946us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:16.775961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:16.776715 (+ 754us) server_negotiation.cc:407] Connection header received
0928 03:04:16.778173 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:16.778182 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:16.778427 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:16.778594 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:16.779616 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.782565 (+ 2949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.786441 (+ 3876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.788014 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.788148 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:16.788412 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:16.788907 (+ 495us) server_negotiation.cc:299] Negotiation successful
0928 03:04:16.789083 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1798,"thread_start_us":1618,"threads_started":1}
I20260928 03:04:16.790254 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:16.773715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:35031 (local address 127.0.0.1:55924)
0928 03:04:16.776540 (+ 2825us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:16.776588 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:16.777964 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:16.778920 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:16.778924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:16.778947 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:16.779364 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:16.779376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.782772 (+ 3396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.782777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:16.786249 (+ 3472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:16.786264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.788439 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.788448 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:16.788661 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:16.788693 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 03:04:16.789722 (+ 1029us) client_negotiation.cc:240] Negotiation successful
0928 03:04:16.789876 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2554,"thread_start_us":2832,"threads_started":1}
I20260928 03:04:16.793133 8177 tablet_service.cc:1467] Processing CreateTablet for tablet 83b48fddc278493e97bd3912f9369df1 (DEFAULT_TABLE table=test-table [id=dc11e1feee31427faeca697fa8e05435]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:16.794800 8177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b48fddc278493e97bd3912f9369df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:16.794337 8456 tablet_service.cc:1467] Processing CreateTablet for tablet 83b48fddc278493e97bd3912f9369df1 (DEFAULT_TABLE table=test-table [id=dc11e1feee31427faeca697fa8e05435]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:16.795411 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:16.777105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:41729 (local address 127.0.0.1:33852)
0928 03:04:16.787740 (+ 10635us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:16.787760 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:16.787874 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:16.789557 (+ 1683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:16.789564 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:16.789593 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:16.789776 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:16.789786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.792024 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.792029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:16.793195 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:16.793207 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.794571 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.794577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:16.794763 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:16.794821 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 03:04:16.794936 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 03:04:16.795126 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10516,"thread_start_us":11047,"threads_started":1}
I20260928 03:04:16.795925 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:16.777381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.6.192.2:41729)
0928 03:04:16.788865 (+ 11484us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:16.788873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:16.788903 (+ 30us) server_negotiation.cc:407] Connection header received
0928 03:04:16.789004 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:16.789012 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:16.789318 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:16.790232 (+ 914us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:16.790479 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.791873 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.793371 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:16.794399 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:16.794558 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:16.794768 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:16.795192 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 03:04:16.795379 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11297,"thread_start_us":11169,"threads_started":1}
I20260928 03:04:16.796873 8456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b48fddc278493e97bd3912f9369df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:16.810041 8545 tablet_bootstrap.cc:492] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0: Bootstrap starting.
I20260928 03:04:16.810565 8317 tablet_service.cc:1467] Processing CreateTablet for tablet 83b48fddc278493e97bd3912f9369df1 (DEFAULT_TABLE table=test-table [id=dc11e1feee31427faeca697fa8e05435]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:16.813877 8317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b48fddc278493e97bd3912f9369df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:16.813854 8546 tablet_bootstrap.cc:492] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800: Bootstrap starting.
I20260928 03:04:16.815219 8545 tablet_bootstrap.cc:654] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:16.816644 8545 log.cc:824] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:16.818097 8546 tablet_bootstrap.cc:654] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:16.819777 8546 log.cc:824] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:16.824725 8545 tablet_bootstrap.cc:492] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0: No bootstrap required, opened a new log
I20260928 03:04:16.825132 8545 ts_tablet_manager.cc:1397] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.003s
I20260928 03:04:16.826339 8546 tablet_bootstrap.cc:492] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800: No bootstrap required, opened a new log
I20260928 03:04:16.826669 8546 ts_tablet_manager.cc:1397] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260928 03:04:16.832360 8549 tablet_bootstrap.cc:492] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b: Bootstrap starting.
I20260928 03:04:16.835836 8545 raft_consensus.cc:348] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } } peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } }
I20260928 03:04:16.836295 8545 raft_consensus.cc:374] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:16.836405 8545 raft_consensus.cc:729] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee9eb7bfd6d4567b10145bb7034cbc0, State: Initialized, Role: FOLLOWER
I20260928 03:04:16.836730 8549 tablet_bootstrap.cc:654] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:16.837083 8545 consensus_queue.cc:260] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } } peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } }
I20260928 03:04:16.838284 8549 log.cc:824] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:16.838466 8546 raft_consensus.cc:348] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } } peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } }
I20260928 03:04:16.839037 8546 raft_consensus.cc:374] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:16.839179 8546 raft_consensus.cc:729] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2ed1a82633e406abcbc0fe46390f800, State: Initialized, Role: FOLLOWER
I20260928 03:04:16.840049 8545 ts_tablet_manager.cc:1428] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260928 03:04:16.840108 8546 consensus_queue.cc:260] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } } peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } }
W20260928 03:04:16.843850 8248 tablet.cc:2369] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:04:16.844033 8549 tablet_bootstrap.cc:492] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b: No bootstrap required, opened a new log
I20260928 03:04:16.844336 8549 ts_tablet_manager.cc:1397] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260928 03:04:16.845419 8526 heartbeater.cc:503] Master 127.6.192.62:35331 was elected leader, sending a full tablet report...
I20260928 03:04:16.846436 8546 ts_tablet_manager.cc:1428] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800: Time spent starting tablet: real 0.020s user 0.016s sys 0.005s
I20260928 03:04:16.853595 8549 raft_consensus.cc:348] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } } peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } }
I20260928 03:04:16.853998 8549 raft_consensus.cc:374] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:16.854065 8549 raft_consensus.cc:729] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f83e90b75d546d08d108940f6c0347b, State: Initialized, Role: FOLLOWER
I20260928 03:04:16.854599 8549 consensus_queue.cc:260] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } } peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } }
I20260928 03:04:16.855832 8387 heartbeater.cc:503] Master 127.6.192.62:35331 was elected leader, sending a full tablet report...
I20260928 03:04:16.856595 8549 ts_tablet_manager.cc:1428] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
W20260928 03:04:16.907754 8388 tablet.cc:2369] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:04:16.941601 8527 tablet.cc:2369] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:04:17.051936 8552 raft_consensus.cc:482] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:04:17.052299 8552 raft_consensus.cc:504] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } } peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } }
I20260928 03:04:17.054363 8552 leader_election.cc:288] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ee9eb7bfd6d4567b10145bb7034cbc0 (127.6.192.1:39393), 5f83e90b75d546d08d108940f6c0347b (127.6.192.2:41729)
I20260928 03:04:17.060648 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.055428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:60979 (local address 127.6.192.1:39393)
0928 03:04:17.055652 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:17.055657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:17.055674 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:04:17.055737 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:17.055741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:17.055808 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:17.055935 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:17.056376 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.057099 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.058734 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.059661 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.059809 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:17.059924 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:17.060254 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 03:04:17.060382 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 03:04:17.060679 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.054721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:39393 (local address 127.6.192.3:60979)
0928 03:04:17.054911 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:17.054932 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:17.055079 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:17.055955 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:17.055960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:17.055988 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:17.056225 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:17.056234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.057257 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.057262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:17.058568 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:17.058579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.059822 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.059828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:17.060045 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:17.060078 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 03:04:17.060197 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 03:04:17.060367 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 03:04:17.061038 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.055515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:44577 (local address 127.6.192.2:41729)
0928 03:04:17.055702 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:17.055709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:17.055869 (+ 160us) server_negotiation.cc:407] Connection header received
0928 03:04:17.056123 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:17.056128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:17.056210 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:17.056325 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:17.056818 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.057521 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.059205 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.060094 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.060244 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:17.060352 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:17.060636 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 03:04:17.060763 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 03:04:17.061136 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.055085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:41729 (local address 127.6.192.3:44577)
0928 03:04:17.055767 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:17.055792 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:17.055934 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:17.056337 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:17.056341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:17.056364 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:17.056678 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:17.056685 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.057659 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.057664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:17.058984 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:17.058994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.060307 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.060313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:17.060484 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:17.060505 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 03:04:17.060612 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 03:04:17.060744 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":990,"threads_started":1}
I20260928 03:04:17.061729 8199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b48fddc278493e97bd3912f9369df1" candidate_uuid: "f2ed1a82633e406abcbc0fe46390f800" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" is_pre_election: true
I20260928 03:04:17.062077 8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b48fddc278493e97bd3912f9369df1" candidate_uuid: "f2ed1a82633e406abcbc0fe46390f800" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f83e90b75d546d08d108940f6c0347b" is_pre_election: true
I20260928 03:04:17.062249 8199 raft_consensus.cc:2454] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2ed1a82633e406abcbc0fe46390f800 in term 0.
I20260928 03:04:17.062542 8337 raft_consensus.cc:2454] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2ed1a82633e406abcbc0fe46390f800 in term 0.
I20260928 03:04:17.062937 8408 leader_election.cc:302] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee9eb7bfd6d4567b10145bb7034cbc0, f2ed1a82633e406abcbc0fe46390f800; no voters:
I20260928 03:04:17.063328 8552 raft_consensus.cc:2781] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:04:17.063580 8552 raft_consensus.cc:482] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:04:17.063695 8552 raft_consensus.cc:3037] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:17.066100 8552 raft_consensus.cc:504] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } } peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } }
I20260928 03:04:17.066680 8552 leader_election.cc:288] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [CANDIDATE]: Term 1 election: Requested vote from peers 3ee9eb7bfd6d4567b10145bb7034cbc0 (127.6.192.1:39393), 5f83e90b75d546d08d108940f6c0347b (127.6.192.2:41729)
I20260928 03:04:17.067260 8199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b48fddc278493e97bd3912f9369df1" candidate_uuid: "f2ed1a82633e406abcbc0fe46390f800" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0"
I20260928 03:04:17.067371 8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b48fddc278493e97bd3912f9369df1" candidate_uuid: "f2ed1a82633e406abcbc0fe46390f800" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f83e90b75d546d08d108940f6c0347b"
I20260928 03:04:17.067523 8199 raft_consensus.cc:3037] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:17.067664 8337 raft_consensus.cc:3037] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:17.069936 8337 raft_consensus.cc:2454] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2ed1a82633e406abcbc0fe46390f800 in term 1.
I20260928 03:04:17.069939 8199 raft_consensus.cc:2454] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2ed1a82633e406abcbc0fe46390f800 in term 1.
I20260928 03:04:17.070379 8408 leader_election.cc:302] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f83e90b75d546d08d108940f6c0347b, f2ed1a82633e406abcbc0fe46390f800; no voters:
I20260928 03:04:17.070701 8552 raft_consensus.cc:2781] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:17.071177 8552 raft_consensus.cc:686] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [term 1 LEADER]: Becoming Leader. State: Replica: f2ed1a82633e406abcbc0fe46390f800, State: Running, Role: LEADER
I20260928 03:04:17.071795 8552 consensus_queue.cc:237] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } } peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } }
I20260928 03:04:17.076613 8039 catalog_manager.cc:5495] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2ed1a82633e406abcbc0fe46390f800 (127.6.192.3). New cstate: current_term: 1 leader_uuid: "f2ed1a82633e406abcbc0fe46390f800" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } health_report { overall_health: UNKNOWN } } }
I20260928 03:04:17.107635 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.101641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55928 (local address 127.6.192.3:35031)
0928 03:04:17.101868 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:17.101877 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:17.102066 (+ 189us) server_negotiation.cc:407] Connection header received
0928 03:04:17.102460 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:17.102464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:17.102548 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:17.102701 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:04:17.103128 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.103925 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.105273 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.105700 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.105835 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:17.106365 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:04:17.106903 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:04:17.107078 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:17.107191 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 03:04:17.107279 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 03:04:17.116371 8337 raft_consensus.cc:1261] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b [term 1 FOLLOWER]: Refusing update from remote peer f2ed1a82633e406abcbc0fe46390f800: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:17.116371 8199 raft_consensus.cc:1261] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 1 FOLLOWER]: Refusing update from remote peer f2ed1a82633e406abcbc0fe46390f800: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:17.117373 8555 consensus_queue.cc:1035] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:04:17.117751 8552 consensus_queue.cc:1035] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:04:17.127059 8558 mvcc.cc:205] Tried to move back new op lower bound from 7334152835537072128 to 7334152835370213376. Current Snapshot: MvccSnapshot[applied={T|T < 7334152835537072128}]
I20260928 03:04:17.136484 8560 mvcc.cc:205] Tried to move back new op lower bound from 7334152835537072128 to 7334152835370213376. Current Snapshot: MvccSnapshot[applied={T|T < 7334152835537072128 or (T in {7334152835537072128})}]
I20260928 03:04:17.152031 8039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:04:17.152367 8039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:04:17.156769 8039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000: Sending DeleteTablet for 3 replicas of tablet 83b48fddc278493e97bd3912f9369df1
I20260928 03:04:17.158042 8456 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b48fddc278493e97bd3912f9369df1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:17 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:55924
I20260928 03:04:17.158092 8177 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b48fddc278493e97bd3912f9369df1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:17 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:33336
I20260928 03:04:17.159548 8317 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b48fddc278493e97bd3912f9369df1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:17 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:33852
I20260928 03:04:17.170544 8571 tablet_replica.cc:330] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0: stopping tablet replica
I20260928 03:04:17.171113 8571 raft_consensus.cc:2229] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:04:17.171669 8571 raft_consensus.cc:2258] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:17.173727 8039 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:04:17.175846 8039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:04:17.176959 8571 ts_tablet_manager.cc:1905] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:17.179961 8570 tablet_replica.cc:330] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800: stopping tablet replica
I20260928 03:04:17.180413 8570 raft_consensus.cc:2229] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:04:17.181268 8570 raft_consensus.cc:2258] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:17.182834 8572 tablet_replica.cc:330] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b: stopping tablet replica
I20260928 03:04:17.183483 8572 raft_consensus.cc:2229] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:04:17.183604 8570 ts_tablet_manager.cc:1905] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:17.184186 8572 raft_consensus.cc:2258] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:17.190521 8177 tablet_service.cc:1467] Processing CreateTablet for tablet f270955a9a3b4ae58896b9675e412077 (DEFAULT_TABLE table=test-table [id=250ef24a7ad6485296626510e2eaae79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:17.191061 8177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f270955a9a3b4ae58896b9675e412077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:17.192370 8317 tablet_service.cc:1467] Processing CreateTablet for tablet f270955a9a3b4ae58896b9675e412077 (DEFAULT_TABLE table=test-table [id=250ef24a7ad6485296626510e2eaae79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:17.192797 8317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f270955a9a3b4ae58896b9675e412077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:17.200052 8572 ts_tablet_manager.cc:1905] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:17.200245 8456 tablet_service.cc:1467] Processing CreateTablet for tablet f270955a9a3b4ae58896b9675e412077 (DEFAULT_TABLE table=test-table [id=250ef24a7ad6485296626510e2eaae79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:17.200747 8456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f270955a9a3b4ae58896b9675e412077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:17.201654 8549 tablet_bootstrap.cc:492] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b: Bootstrap starting.
I20260928 03:04:17.203634 8549 tablet_bootstrap.cc:654] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:17.205796 8571 ts_tablet_manager.cc:1918] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:04:17.208063 8571 log.cc:1196] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/83b48fddc278493e97bd3912f9369df1
I20260928 03:04:17.208420 8546 tablet_bootstrap.cc:492] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800: Bootstrap starting.
I20260928 03:04:17.208716 8545 tablet_bootstrap.cc:492] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0: Bootstrap starting.
I20260928 03:04:17.209120 8571 ts_tablet_manager.cc:1939] T 83b48fddc278493e97bd3912f9369df1 P 3ee9eb7bfd6d4567b10145bb7034cbc0: Deleting consensus metadata
I20260928 03:04:17.210351 8546 tablet_bootstrap.cc:654] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:17.210508 8545 tablet_bootstrap.cc:654] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:17.211129 8549 tablet_bootstrap.cc:492] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b: No bootstrap required, opened a new log
I20260928 03:04:17.211295 8549 ts_tablet_manager.cc:1397] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20260928 03:04:17.212267 8549 raft_consensus.cc:348] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } }
I20260928 03:04:17.212525 8549 raft_consensus.cc:374] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:17.212638 8549 raft_consensus.cc:729] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f83e90b75d546d08d108940f6c0347b, State: Initialized, Role: FOLLOWER
I20260928 03:04:17.212941 8549 consensus_queue.cc:260] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } }
I20260928 03:04:17.213771 8549 ts_tablet_manager.cc:1428] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 03:04:17.215957 8570 ts_tablet_manager.cc:1918] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:04:17.215969 8025 catalog_manager.cc:4841] TS 3ee9eb7bfd6d4567b10145bb7034cbc0 (127.6.192.1:39393): tablet 83b48fddc278493e97bd3912f9369df1 (table test-table [id=dc11e1feee31427faeca697fa8e05435]) successfully deleted
I20260928 03:04:17.218317 8572 ts_tablet_manager.cc:1918] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:04:17.218735 8570 log.cc:1196] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/83b48fddc278493e97bd3912f9369df1
I20260928 03:04:17.219370 8570 ts_tablet_manager.cc:1939] T 83b48fddc278493e97bd3912f9369df1 P f2ed1a82633e406abcbc0fe46390f800: Deleting consensus metadata
I20260928 03:04:17.221560 8572 log.cc:1196] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/83b48fddc278493e97bd3912f9369df1
I20260928 03:04:17.221603 8024 catalog_manager.cc:4841] TS f2ed1a82633e406abcbc0fe46390f800 (127.6.192.3:35031): tablet 83b48fddc278493e97bd3912f9369df1 (table test-table [id=dc11e1feee31427faeca697fa8e05435]) successfully deleted
I20260928 03:04:17.222257 8572 ts_tablet_manager.cc:1939] T 83b48fddc278493e97bd3912f9369df1 P 5f83e90b75d546d08d108940f6c0347b: Deleting consensus metadata
I20260928 03:04:17.223251 8546 tablet_bootstrap.cc:492] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800: No bootstrap required, opened a new log
I20260928 03:04:17.223428 8545 tablet_bootstrap.cc:492] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0: No bootstrap required, opened a new log
I20260928 03:04:17.223502 8546 ts_tablet_manager.cc:1397] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.013s
I20260928 03:04:17.223624 8545 ts_tablet_manager.cc:1397] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20260928 03:04:17.224282 8025 catalog_manager.cc:4841] TS 5f83e90b75d546d08d108940f6c0347b (127.6.192.2:41729): tablet 83b48fddc278493e97bd3912f9369df1 (table test-table [id=dc11e1feee31427faeca697fa8e05435]) successfully deleted
I20260928 03:04:17.224545 8546 raft_consensus.cc:348] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } }
I20260928 03:04:17.224532 8545 raft_consensus.cc:348] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } }
I20260928 03:04:17.224802 8546 raft_consensus.cc:374] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:17.224793 8545 raft_consensus.cc:374] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:17.224913 8545 raft_consensus.cc:729] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee9eb7bfd6d4567b10145bb7034cbc0, State: Initialized, Role: FOLLOWER
I20260928 03:04:17.224925 8546 raft_consensus.cc:729] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2ed1a82633e406abcbc0fe46390f800, State: Initialized, Role: FOLLOWER
I20260928 03:04:17.225230 8546 consensus_queue.cc:260] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } }
I20260928 03:04:17.225330 8545 consensus_queue.cc:260] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } }
I20260928 03:04:17.225970 8545 ts_tablet_manager.cc:1428] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 03:04:17.226639 8546 ts_tablet_manager.cc:1428] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 03:04:17.418819 8553 raft_consensus.cc:482] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:04:17.419078 8553 raft_consensus.cc:504] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } }
I20260928 03:04:17.421061 8553 leader_election.cc:288] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ee9eb7bfd6d4567b10145bb7034cbc0 (127.6.192.1:39393), f2ed1a82633e406abcbc0fe46390f800 (127.6.192.3:35031)
I20260928 03:04:17.430316 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.421535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:53717 (local address 127.6.192.1:39393)
0928 03:04:17.421730 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:17.421736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:17.422762 (+ 1026us) server_negotiation.cc:407] Connection header received
0928 03:04:17.423106 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:17.423112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:17.423208 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:17.423275 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:17.424333 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.425420 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.427813 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.429144 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.429290 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:17.429416 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:17.429853 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 03:04:17.430025 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 03:04:17.430563 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.421706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:41765 (local address 127.6.192.3:35031)
0928 03:04:17.421882 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:17.421888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:17.423015 (+ 1127us) server_negotiation.cc:407] Connection header received
0928 03:04:17.423219 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:17.423224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:17.423282 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:17.423388 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:17.424056 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.425298 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.428125 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.429475 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.429632 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:17.429731 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:17.430093 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 03:04:17.430223 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 03:04:17.430897 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.421390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:39393 (local address 127.6.192.2:53717)
0928 03:04:17.422618 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:17.422640 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:17.422910 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:17.423591 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:17.423597 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:17.423632 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:17.424149 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:17.424162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.425556 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.425562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:17.427647 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:17.427663 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.429401 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.429409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:17.429616 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:17.429660 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 03:04:17.429817 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 03:04:17.429974 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1056,"mutex_wait_us":150,"thread_start_us":704,"threads_started":1}
I20260928 03:04:17.431391 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.421428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:35031 (local address 127.6.192.2:41765)
0928 03:04:17.422890 (+ 1462us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:17.422909 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:17.423068 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:17.423421 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:17.423426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:17.423480 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:17.423872 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:17.423881 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.425474 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.425479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:17.427962 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:17.427978 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.429695 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.429701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:17.429869 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:17.429899 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 03:04:17.430749 (+ 850us) client_negotiation.cc:240] Negotiation successful
0928 03:04:17.430926 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1359,"mutex_wait_us":126,"thread_start_us":1184,"threads_started":1}
I20260928 03:04:17.431775 8199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f270955a9a3b4ae58896b9675e412077" candidate_uuid: "5f83e90b75d546d08d108940f6c0347b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" is_pre_election: true
I20260928 03:04:17.432085 8199 raft_consensus.cc:2454] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f83e90b75d546d08d108940f6c0347b in term 0.
I20260928 03:04:17.432582 8268 leader_election.cc:302] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee9eb7bfd6d4567b10145bb7034cbc0, 5f83e90b75d546d08d108940f6c0347b; no voters:
I20260928 03:04:17.432591 8476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f270955a9a3b4ae58896b9675e412077" candidate_uuid: "5f83e90b75d546d08d108940f6c0347b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2ed1a82633e406abcbc0fe46390f800" is_pre_election: true
I20260928 03:04:17.432950 8553 raft_consensus.cc:2781] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:04:17.433066 8553 raft_consensus.cc:482] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:04:17.433149 8553 raft_consensus.cc:3037] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:17.433065 8476 raft_consensus.cc:2454] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f83e90b75d546d08d108940f6c0347b in term 0.
I20260928 03:04:17.436177 8553 raft_consensus.cc:504] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } }
I20260928 03:04:17.436905 8553 leader_election.cc:288] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [CANDIDATE]: Term 1 election: Requested vote from peers 3ee9eb7bfd6d4567b10145bb7034cbc0 (127.6.192.1:39393), f2ed1a82633e406abcbc0fe46390f800 (127.6.192.3:35031)
I20260928 03:04:17.437490 8199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f270955a9a3b4ae58896b9675e412077" candidate_uuid: "5f83e90b75d546d08d108940f6c0347b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0"
I20260928 03:04:17.437515 8476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f270955a9a3b4ae58896b9675e412077" candidate_uuid: "5f83e90b75d546d08d108940f6c0347b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2ed1a82633e406abcbc0fe46390f800"
I20260928 03:04:17.437738 8476 raft_consensus.cc:3037] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:17.437734 8199 raft_consensus.cc:3037] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:17.441018 8199 raft_consensus.cc:2454] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f83e90b75d546d08d108940f6c0347b in term 1.
I20260928 03:04:17.441186 8476 raft_consensus.cc:2454] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f83e90b75d546d08d108940f6c0347b in term 1.
I20260928 03:04:17.441726 8268 leader_election.cc:302] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee9eb7bfd6d4567b10145bb7034cbc0, 5f83e90b75d546d08d108940f6c0347b; no voters:
I20260928 03:04:17.442107 8553 raft_consensus.cc:2781] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:17.442647 8553 raft_consensus.cc:686] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [term 1 LEADER]: Becoming Leader. State: Replica: 5f83e90b75d546d08d108940f6c0347b, State: Running, Role: LEADER
I20260928 03:04:17.443084 8553 consensus_queue.cc:237] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } }
I20260928 03:04:17.446910 8040 catalog_manager.cc:5495] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f83e90b75d546d08d108940f6c0347b (127.6.192.2). New cstate: current_term: 1 leader_uuid: "5f83e90b75d546d08d108940f6c0347b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } health_report { overall_health: UNKNOWN } } }
Sep 28 03:04:17 dist-test-slave-fjww krb5kdc[7995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564654, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:17.552098 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.530438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:35331 (local address 127.0.0.1:58566)
0928 03:04:17.531551 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:17.531584 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:17.531798 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:17.532188 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:17.532195 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:17.532975 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:17.533201 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:17.533212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.534511 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.534517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:17.535684 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:17.535698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.536491 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.536497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:17.536701 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:17.543348 (+ 6647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:17.543377 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:17.545940 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:17.549325 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:17.549341 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:17.549350 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:17.549800 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:17.550203 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:17.550209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:17.550212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:17.550353 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:17.550938 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:17.550947 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:17.551145 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 03:04:17.551315 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 03:04:17.551652 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":607,"threads_started":1}
I20260928 03:04:17.552161 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.530477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58566 (local address 127.6.192.62:35331)
0928 03:04:17.531868 (+ 1391us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:17.531877 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:17.531909 (+ 32us) server_negotiation.cc:407] Connection header received
0928 03:04:17.531962 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:17.531969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:17.532031 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:17.532155 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:17.533377 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.534362 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.535839 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.536310 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.536467 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:17.546202 (+ 9735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:17.546207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:17.546210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:17.546241 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:17.549158 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:17.549950 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:17.549956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:17.549961 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:17.550089 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:17.550471 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:17.550476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:17.550479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:17.550789 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:17.550983 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:17.551420 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 03:04:17.551791 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":1242,"thread_start_us":1075,"threads_started":1}
I20260928 03:04:17.558233 8040 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:58566:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 03:04:17.559376 8040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:04:17.567023 8177 tablet_service.cc:1467] Processing CreateTablet for tablet e206c015f74144268e2959ddceb0676d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=698ee51727d34965ab4b18333c007ac7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 03:04:17.567065 8317 tablet_service.cc:1467] Processing CreateTablet for tablet e206c015f74144268e2959ddceb0676d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=698ee51727d34965ab4b18333c007ac7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 03:04:17.567669 8317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e206c015f74144268e2959ddceb0676d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:17.567698 8177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e206c015f74144268e2959ddceb0676d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:17.572054 8456 tablet_service.cc:1467] Processing CreateTablet for tablet e206c015f74144268e2959ddceb0676d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=698ee51727d34965ab4b18333c007ac7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 03:04:17.572635 8456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e206c015f74144268e2959ddceb0676d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:17.574450 8545 tablet_bootstrap.cc:492] T e206c015f74144268e2959ddceb0676d P 3ee9eb7bfd6d4567b10145bb7034cbc0: Bootstrap starting.
I20260928 03:04:17.576001 8549 tablet_bootstrap.cc:492] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b: Bootstrap starting.
I20260928 03:04:17.576174 8545 tablet_bootstrap.cc:654] T e206c015f74144268e2959ddceb0676d P 3ee9eb7bfd6d4567b10145bb7034cbc0: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:17.577747 8549 tablet_bootstrap.cc:654] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:17.586722 8546 tablet_bootstrap.cc:492] T e206c015f74144268e2959ddceb0676d P f2ed1a82633e406abcbc0fe46390f800: Bootstrap starting.
I20260928 03:04:17.588173 8549 tablet_bootstrap.cc:492] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b: No bootstrap required, opened a new log
I20260928 03:04:17.588405 8549 ts_tablet_manager.cc:1397] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.009s
I20260928 03:04:17.588723 8546 tablet_bootstrap.cc:654] T e206c015f74144268e2959ddceb0676d P f2ed1a82633e406abcbc0fe46390f800: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:17.589183 8549 raft_consensus.cc:348] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } }
I20260928 03:04:17.589445 8549 raft_consensus.cc:374] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:17.589529 8549 raft_consensus.cc:729] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f83e90b75d546d08d108940f6c0347b, State: Initialized, Role: FOLLOWER
I20260928 03:04:17.589784 8549 consensus_queue.cc:260] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } }
I20260928 03:04:17.590238 8553 tablet_replica.cc:439] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } } }
I20260928 03:04:17.590526 8553 tablet_replica.cc:442] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:04:17.591500 8549 ts_tablet_manager.cc:1428] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 03:04:17.603024 8546 tablet_bootstrap.cc:492] T e206c015f74144268e2959ddceb0676d P f2ed1a82633e406abcbc0fe46390f800: No bootstrap required, opened a new log
I20260928 03:04:17.603221 8546 ts_tablet_manager.cc:1397] T e206c015f74144268e2959ddceb0676d P f2ed1a82633e406abcbc0fe46390f800: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
I20260928 03:04:17.604238 8545 tablet_bootstrap.cc:492] T e206c015f74144268e2959ddceb0676d P 3ee9eb7bfd6d4567b10145bb7034cbc0: No bootstrap required, opened a new log
I20260928 03:04:17.604140 8546 raft_consensus.cc:348] T e206c015f74144268e2959ddceb0676d P f2ed1a82633e406abcbc0fe46390f800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } }
I20260928 03:04:17.604435 8545 ts_tablet_manager.cc:1397] T e206c015f74144268e2959ddceb0676d P 3ee9eb7bfd6d4567b10145bb7034cbc0: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.008s
I20260928 03:04:17.604444 8546 raft_consensus.cc:374] T e206c015f74144268e2959ddceb0676d P f2ed1a82633e406abcbc0fe46390f800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:17.604521 8546 raft_consensus.cc:729] T e206c015f74144268e2959ddceb0676d P f2ed1a82633e406abcbc0fe46390f800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2ed1a82633e406abcbc0fe46390f800, State: Initialized, Role: FOLLOWER
I20260928 03:04:17.604840 8546 consensus_queue.cc:260] T e206c015f74144268e2959ddceb0676d P f2ed1a82633e406abcbc0fe46390f800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } }
I20260928 03:04:17.605252 8545 raft_consensus.cc:348] T e206c015f74144268e2959ddceb0676d P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } }
I20260928 03:04:17.605533 8545 raft_consensus.cc:374] T e206c015f74144268e2959ddceb0676d P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:17.605300 8566 tablet_replica.cc:439] T e206c015f74144268e2959ddceb0676d P f2ed1a82633e406abcbc0fe46390f800: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } } }
I20260928 03:04:17.605623 8545 raft_consensus.cc:729] T e206c015f74144268e2959ddceb0676d P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee9eb7bfd6d4567b10145bb7034cbc0, State: Initialized, Role: FOLLOWER
I20260928 03:04:17.605666 8566 tablet_replica.cc:442] T e206c015f74144268e2959ddceb0676d P f2ed1a82633e406abcbc0fe46390f800: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:04:17.605954 8545 consensus_queue.cc:260] T e206c015f74144268e2959ddceb0676d P 3ee9eb7bfd6d4567b10145bb7034cbc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } }
I20260928 03:04:17.606698 8546 ts_tablet_manager.cc:1428] T e206c015f74144268e2959ddceb0676d P f2ed1a82633e406abcbc0fe46390f800: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 03:04:17.606436 8550 tablet_replica.cc:439] T e206c015f74144268e2959ddceb0676d P 3ee9eb7bfd6d4567b10145bb7034cbc0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } } }
I20260928 03:04:17.606846 8550 tablet_replica.cc:442] T e206c015f74144268e2959ddceb0676d P 3ee9eb7bfd6d4567b10145bb7034cbc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:04:17.607651 8545 ts_tablet_manager.cc:1428] T e206c015f74144268e2959ddceb0676d P 3ee9eb7bfd6d4567b10145bb7034cbc0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 03:04:17.644042 8553 raft_consensus.cc:482] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:04:17.644300 8553 raft_consensus.cc:504] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } }
I20260928 03:04:17.645104 8553 leader_election.cc:288] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ee9eb7bfd6d4567b10145bb7034cbc0 (127.6.192.1:39393), f2ed1a82633e406abcbc0fe46390f800 (127.6.192.3:35031)
I20260928 03:04:17.645630 8199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e206c015f74144268e2959ddceb0676d" candidate_uuid: "5f83e90b75d546d08d108940f6c0347b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" is_pre_election: true
I20260928 03:04:17.645699 8476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e206c015f74144268e2959ddceb0676d" candidate_uuid: "5f83e90b75d546d08d108940f6c0347b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2ed1a82633e406abcbc0fe46390f800" is_pre_election: true
I20260928 03:04:17.645886 8199 raft_consensus.cc:2454] T e206c015f74144268e2959ddceb0676d P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f83e90b75d546d08d108940f6c0347b in term 0.
I20260928 03:04:17.646008 8476 raft_consensus.cc:2454] T e206c015f74144268e2959ddceb0676d P f2ed1a82633e406abcbc0fe46390f800 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f83e90b75d546d08d108940f6c0347b in term 0.
I20260928 03:04:17.646376 8268 leader_election.cc:302] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee9eb7bfd6d4567b10145bb7034cbc0, 5f83e90b75d546d08d108940f6c0347b; no voters:
I20260928 03:04:17.646669 8553 raft_consensus.cc:2781] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:04:17.646751 8553 raft_consensus.cc:482] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:04:17.646839 8553 raft_consensus.cc:3037] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:17.649026 8553 raft_consensus.cc:504] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } }
I20260928 03:04:17.649636 8553 leader_election.cc:288] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b [CANDIDATE]: Term 1 election: Requested vote from peers 3ee9eb7bfd6d4567b10145bb7034cbc0 (127.6.192.1:39393), f2ed1a82633e406abcbc0fe46390f800 (127.6.192.3:35031)
I20260928 03:04:17.650079 8476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e206c015f74144268e2959ddceb0676d" candidate_uuid: "5f83e90b75d546d08d108940f6c0347b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2ed1a82633e406abcbc0fe46390f800"
I20260928 03:04:17.650079 8199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e206c015f74144268e2959ddceb0676d" candidate_uuid: "5f83e90b75d546d08d108940f6c0347b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0"
I20260928 03:04:17.650324 8476 raft_consensus.cc:3037] T e206c015f74144268e2959ddceb0676d P f2ed1a82633e406abcbc0fe46390f800 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:17.650328 8199 raft_consensus.cc:3037] T e206c015f74144268e2959ddceb0676d P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:17.653288 8199 raft_consensus.cc:2454] T e206c015f74144268e2959ddceb0676d P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f83e90b75d546d08d108940f6c0347b in term 1.
I20260928 03:04:17.653278 8476 raft_consensus.cc:2454] T e206c015f74144268e2959ddceb0676d P f2ed1a82633e406abcbc0fe46390f800 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f83e90b75d546d08d108940f6c0347b in term 1.
I20260928 03:04:17.653846 8267 leader_election.cc:302] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f83e90b75d546d08d108940f6c0347b, f2ed1a82633e406abcbc0fe46390f800; no voters:
I20260928 03:04:17.654177 8553 raft_consensus.cc:2781] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:17.654386 8553 raft_consensus.cc:686] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b [term 1 LEADER]: Becoming Leader. State: Replica: 5f83e90b75d546d08d108940f6c0347b, State: Running, Role: LEADER
I20260928 03:04:17.654696 8553 consensus_queue.cc:237] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } }
I20260928 03:04:17.656563 8578 tablet_replica.cc:439] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b: TxnStatusTablet state changed. Reason: New leader 5f83e90b75d546d08d108940f6c0347b. Latest consensus state: current_term: 1 leader_uuid: "5f83e90b75d546d08d108940f6c0347b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } } }
I20260928 03:04:17.656908 8578 tablet_replica.cc:442] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b: This TxnStatusTablet replica's current role is: LEADER
I20260928 03:04:17.657593 8591 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 03:04:17.657828 8591 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 03:04:17.658248 8040 catalog_manager.cc:5495] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f83e90b75d546d08d108940f6c0347b (127.6.192.2). New cstate: current_term: 1 leader_uuid: "5f83e90b75d546d08d108940f6c0347b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } health_report { overall_health: UNKNOWN } } }
I20260928 03:04:17.698702 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.690005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33854 (local address 127.6.192.2:41729)
0928 03:04:17.691321 (+ 1316us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:17.691328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:17.691353 (+ 25us) server_negotiation.cc:407] Connection header received
0928 03:04:17.691500 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:17.691506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:17.691632 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:17.691819 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:04:17.692266 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.693627 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.695668 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.696382 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.696522 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:17.697043 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:04:17.697940 (+ 897us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:04:17.698077 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:17.698223 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 03:04:17.698321 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1147,"thread_start_us":968,"threads_started":1}
I20260928 03:04:17.698732 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.690029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:41729 (local address 127.0.0.1:33854)
0928 03:04:17.690449 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:17.690471 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:17.690631 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:17.691860 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:17.691865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:17.691881 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 03:04:17.692086 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:17.692096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.693794 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.693800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:17.695505 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:17.695522 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.696552 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.696558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:17.696757 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:17.696834 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 03:04:17.698076 (+ 1242us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 03:04:17.698103 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 03:04:17.698183 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 03:04:17.698386 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238}
I20260928 03:04:17.704633 8476 raft_consensus.cc:1261] T e206c015f74144268e2959ddceb0676d P f2ed1a82633e406abcbc0fe46390f800 [term 1 FOLLOWER]: Refusing update from remote peer 5f83e90b75d546d08d108940f6c0347b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:17.704633 8199 raft_consensus.cc:1261] T e206c015f74144268e2959ddceb0676d P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 1 FOLLOWER]: Refusing update from remote peer 5f83e90b75d546d08d108940f6c0347b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:17.705518 8578 consensus_queue.cc:1035] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:04:17.705952 8553 consensus_queue.cc:1035] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:04:17.726509 8566 tablet_replica.cc:439] T e206c015f74144268e2959ddceb0676d P f2ed1a82633e406abcbc0fe46390f800: TxnStatusTablet state changed. Reason: New leader 5f83e90b75d546d08d108940f6c0347b. Latest consensus state: current_term: 1 leader_uuid: "5f83e90b75d546d08d108940f6c0347b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } } }
I20260928 03:04:17.726871 8566 tablet_replica.cc:442] T e206c015f74144268e2959ddceb0676d P f2ed1a82633e406abcbc0fe46390f800: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:04:17.727857 8550 tablet_replica.cc:439] T e206c015f74144268e2959ddceb0676d P 3ee9eb7bfd6d4567b10145bb7034cbc0: TxnStatusTablet state changed. Reason: New leader 5f83e90b75d546d08d108940f6c0347b. Latest consensus state: current_term: 1 leader_uuid: "5f83e90b75d546d08d108940f6c0347b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } } }
I20260928 03:04:17.728185 8550 tablet_replica.cc:442] T e206c015f74144268e2959ddceb0676d P 3ee9eb7bfd6d4567b10145bb7034cbc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:04:17.735663 8553 tablet_replica.cc:439] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f83e90b75d546d08d108940f6c0347b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } } }
I20260928 03:04:17.736008 8553 tablet_replica.cc:442] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b: This TxnStatusTablet replica's current role is: LEADER
I20260928 03:04:17.739620 8553 tablet_replica.cc:439] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f83e90b75d546d08d108940f6c0347b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } } }
I20260928 03:04:17.739951 8553 tablet_replica.cc:442] T e206c015f74144268e2959ddceb0676d P 5f83e90b75d546d08d108940f6c0347b: This TxnStatusTablet replica's current role is: LEADER
I20260928 03:04:17.749230 8566 tablet_replica.cc:439] T e206c015f74144268e2959ddceb0676d P f2ed1a82633e406abcbc0fe46390f800: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f83e90b75d546d08d108940f6c0347b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } } }
I20260928 03:04:17.749519 8566 tablet_replica.cc:442] T e206c015f74144268e2959ddceb0676d P f2ed1a82633e406abcbc0fe46390f800: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:04:17.751003 8550 tablet_replica.cc:439] T e206c015f74144268e2959ddceb0676d P 3ee9eb7bfd6d4567b10145bb7034cbc0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f83e90b75d546d08d108940f6c0347b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 } } peers { permanent_uuid: "5f83e90b75d546d08d108940f6c0347b" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 41729 } } peers { permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 } } }
I20260928 03:04:17.751273 8550 tablet_replica.cc:442] T e206c015f74144268e2959ddceb0676d P 3ee9eb7bfd6d4567b10145bb7034cbc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:04:17.762584 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.749745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33870 (local address 127.6.192.2:41729)
0928 03:04:17.750048 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:17.750054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:17.756239 (+ 6185us) server_negotiation.cc:407] Connection header received
0928 03:04:17.756404 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:17.756410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:17.756505 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:17.756615 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:04:17.757170 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.758343 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.760397 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.761148 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.761296 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:17.761731 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:04:17.761876 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:04:17.761977 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:17.762137 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 03:04:17.762226 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260928 03:04:17.783025 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.775360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:60133 (local address 127.6.192.2:41729)
0928 03:04:17.775643 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:17.775648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:17.776424 (+ 776us) server_negotiation.cc:407] Connection header received
0928 03:04:17.776672 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:17.776682 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:17.776779 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:17.776921 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:04:17.777603 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.778588 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.780892 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.781445 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.781592 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:17.782104 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:04:17.782220 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:04:17.782363 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:17.782504 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 03:04:17.782581 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":136}
I20260928 03:04:17.783367 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.775397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:41729 (local address 127.6.192.2:60133)
0928 03:04:17.776258 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:17.776284 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:17.776526 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:17.776935 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:17.776940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:17.776951 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 03:04:17.777435 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:17.777446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.778773 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.778799 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:17.780711 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:17.780740 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.781675 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.781682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:17.781836 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:17.781897 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 03:04:17.782375 (+ 478us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 03:04:17.782388 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 03:04:17.782471 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 03:04:17.782627 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":42,"thread_start_us":523,"threads_started":1}
I20260928 03:04:17.793952 8199 raft_consensus.cc:1261] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 1 FOLLOWER]: Refusing update from remote peer 5f83e90b75d546d08d108940f6c0347b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:17.794379 8476 raft_consensus.cc:1261] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800 [term 1 FOLLOWER]: Refusing update from remote peer 5f83e90b75d546d08d108940f6c0347b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:17.794644 8608 consensus_queue.cc:1035] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee9eb7bfd6d4567b10145bb7034cbc0" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 39393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:04:17.795106 8578 consensus_queue.cc:1035] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2ed1a82633e406abcbc0fe46390f800" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:04:17.880380 8040 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:04:17.880728 8040 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:04:17.884059 8040 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d3fa10d7bea40d49b73dd5944aec000: Sending DeleteTablet for 3 replicas of tablet f270955a9a3b4ae58896b9675e412077
I20260928 03:04:17.885072 8456 tablet_service.cc:1514] Processing DeleteTablet for tablet f270955a9a3b4ae58896b9675e412077 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:17 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:55924
I20260928 03:04:17.885529 8177 tablet_service.cc:1514] Processing DeleteTablet for tablet f270955a9a3b4ae58896b9675e412077 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:17 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:33336
I20260928 03:04:17.885885 8613 tablet_replica.cc:330] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800: stopping tablet replica
I20260928 03:04:17.886150 8613 raft_consensus.cc:2229] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:04:17.886277 8317 tablet_service.cc:1514] Processing DeleteTablet for tablet f270955a9a3b4ae58896b9675e412077 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:17 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:33852
I20260928 03:04:17.886634 8613 raft_consensus.cc:2258] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:17.888190 8613 ts_tablet_manager.cc:1905] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:17.892469 8614 tablet_replica.cc:330] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0: stopping tablet replica
I20260928 03:04:17.892704 8614 raft_consensus.cc:2229] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:04:17.893005 8614 raft_consensus.cc:2258] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:17.895169 8614 ts_tablet_manager.cc:1905] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:17.897879 8613 ts_tablet_manager.cc:1918] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 03:04:17.898115 8613 log.cc:1196] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/f270955a9a3b4ae58896b9675e412077
I20260928 03:04:17.898769 8613 ts_tablet_manager.cc:1939] T f270955a9a3b4ae58896b9675e412077 P f2ed1a82633e406abcbc0fe46390f800: Deleting consensus metadata
I20260928 03:04:17.899650 8616 tablet_replica.cc:330] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b: stopping tablet replica
I20260928 03:04:17.899873 8616 raft_consensus.cc:2229] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:04:17.900287 8024 catalog_manager.cc:4841] TS f2ed1a82633e406abcbc0fe46390f800 (127.6.192.3:35031): tablet f270955a9a3b4ae58896b9675e412077 (table test-table [id=250ef24a7ad6485296626510e2eaae79]) successfully deleted
I20260928 03:04:17.900530 8616 raft_consensus.cc:2258] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:17.901611 8616 ts_tablet_manager.cc:1905] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:17.904742 8614 ts_tablet_manager.cc:1918] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 03:04:17.904906 8614 log.cc:1196] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/f270955a9a3b4ae58896b9675e412077
I20260928 03:04:17.905418 8614 ts_tablet_manager.cc:1939] T f270955a9a3b4ae58896b9675e412077 P 3ee9eb7bfd6d4567b10145bb7034cbc0: Deleting consensus metadata
I20260928 03:04:17.907234 8025 catalog_manager.cc:4841] TS 3ee9eb7bfd6d4567b10145bb7034cbc0 (127.6.192.1:39393): tablet f270955a9a3b4ae58896b9675e412077 (table test-table [id=250ef24a7ad6485296626510e2eaae79]) successfully deleted
I20260928 03:04:17.909684 8616 ts_tablet_manager.cc:1918] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 03:04:17.909847 8616 log.cc:1196] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/f270955a9a3b4ae58896b9675e412077
I20260928 03:04:17.910365 8616 ts_tablet_manager.cc:1939] T f270955a9a3b4ae58896b9675e412077 P 5f83e90b75d546d08d108940f6c0347b: Deleting consensus metadata
I20260928 03:04:17.911660 8025 catalog_manager.cc:4841] TS 5f83e90b75d546d08d108940f6c0347b (127.6.192.2:41729): tablet f270955a9a3b4ae58896b9675e412077 (table test-table [id=250ef24a7ad6485296626510e2eaae79]) successfully deleted
Sep 28 03:04:17 dist-test-slave-fjww krb5kdc[7995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564656, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:04:17.924062 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.898949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.6.192.1:39393)
0928 03:04:17.899156 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:17.899162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:17.904137 (+ 4975us) server_negotiation.cc:407] Connection header received
0928 03:04:17.904456 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:17.904462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:17.904567 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:17.904646 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:17.907801 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.908568 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.910259 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.910820 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.910903 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:17.916567 (+ 5664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:17.916583 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:17.916601 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:17.916656 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:17.920220 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:17.921189 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:17.921202 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:17.921211 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:17.921455 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:17.921956 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:17.921962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:17.921966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:17.922426 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:17.922820 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:17.923196 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 03:04:17.923509 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 03:04:17.924988 8242 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33348
I20260928 03:04:17.955240 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.932042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58580 (local address 127.6.192.62:35331)
0928 03:04:17.932364 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:17.932375 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:17.933038 (+ 663us) server_negotiation.cc:407] Connection header received
0928 03:04:17.933327 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:17.933336 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:17.933426 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:17.933566 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:17.935220 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.936249 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.938469 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.939526 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.939702 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:17.945412 (+ 5710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:17.945424 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:17.945431 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:17.945494 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:17.950593 (+ 5099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:17.952004 (+ 1411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:17.952016 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:17.952022 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:17.952251 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:17.952912 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:17.952919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:17.952926 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:17.953441 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:17.953760 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:17.954583 (+ 823us) server_negotiation.cc:299] Negotiation successful
0928 03:04:17.954905 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
W20260928 03:04:17.956202 8040 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58580
I20260928 03:04:17.958861 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 8111
I20260928 03:04:17.977193 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.960837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33364 (local address 127.6.192.1:39393)
0928 03:04:17.961067 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:17.961076 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:17.961606 (+ 530us) server_negotiation.cc:407] Connection header received
0928 03:04:17.961857 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:17.961864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:17.961956 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:17.962085 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:17.963538 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.964588 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.966332 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:17.966996 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:17.967173 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:17.970035 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:17.970042 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:17.970046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:17.970089 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:17.973600 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:17.974473 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:17.974480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:17.974492 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:17.974682 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:17.975186 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:17.975194 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:17.975198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:17.975629 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:17.975959 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:17.976349 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 03:04:17.976611 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 03:04:17.978014 8242 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33364
W20260928 03:04:17.978757 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 8111: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:04:17.979022 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 8111
I20260928 03:04:17.998624 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 8251
Sep 28 03:04:18 dist-test-slave-fjww krb5kdc[7995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564656, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.192.2@KRBTEST.COM
I20260928 03:04:18.017030 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:17.999528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33884 (local address 127.6.192.2:41729)
0928 03:04:17.999796 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:17.999800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:18.000118 (+ 318us) server_negotiation.cc:407] Connection header received
0928 03:04:18.000354 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:18.000361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:18.000444 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:18.000572 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:18.001847 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:18.002669 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:18.004126 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:18.004663 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:18.004829 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:18.009455 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:18.009470 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:18.009486 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:18.009538 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:18.013193 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:18.014091 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:18.014099 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:18.014101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:18.014351 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:18.014845 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:18.014857 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:18.014860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:18.015302 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:18.015724 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:18.016222 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 03:04:18.016514 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
W20260928 03:04:18.017912 8382 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33884
W20260928 03:04:18.018740 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 8251: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:04:18.018945 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 8251
I20260928 03:04:18.043017 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 8391
Sep 28 03:04:18 dist-test-slave-fjww krb5kdc[7995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564656, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.192.3@KRBTEST.COM
I20260928 03:04:18.060250 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:18.043854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55930 (local address 127.6.192.3:35031)
0928 03:04:18.044363 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:18.044368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:18.044385 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:04:18.044479 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:18.044483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:18.044545 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:18.044638 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:18.045871 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:18.046616 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:18.047888 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:18.048298 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:18.048395 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:18.052846 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:18.052863 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:18.052879 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:18.052927 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:18.056889 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:18.057974 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:18.057984 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:18.057988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:18.058139 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:18.058567 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:18.058573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:18.058577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:18.058961 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:18.059261 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:18.059602 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 03:04:18.059844 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":290,"threads_started":1}
W20260928 03:04:18.060988 8521 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55930
W20260928 03:04:18.061758 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 8391: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:04:18.061929 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 8391
I20260928 03:04:18.083897 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 8008
I20260928 03:04:18.085194 8080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:18.201790 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 8008
2026-09-28T03:04:18Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5023 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:04:18 dist-test-slave-fjww krb5kdc[8625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:04:18 dist-test-slave-fjww krb5kdc[8625](info): set up 2 sockets
Sep 28 03:04:18 dist-test-slave-fjww krb5kdc[8625](info): commencing operation
krb5kdc: starting...
W20260928 03:04:19.324685 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:04:19 dist-test-slave-fjww krb5kdc[8625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:04:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:04:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:04:19.540124 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:38929
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:38929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:19.876989 8638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:19.877368 8638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:19.877555 8638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:19.885020 8638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:04:19.885147 8638 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 03:04:19.885232 8638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:19.885340 8638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:19.885428 8638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:04:19.885541 8638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:04:19.896201 8638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:38929
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:38929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:19.897583 8638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:19.899652 8638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:19.912386 8644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:19.912542 8646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:19.912556 8643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:19.913777 8638 server_base.cc:1034] running on GCE node
I20260928 03:04:19.914649 8638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:19.916265 8638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:19.917520 8638 hybrid_clock.cc:648] HybridClock initialized: now 1790564659917467 us; error 62 us; skew 500 ppm
I20260928 03:04:19.917867 8638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:19 dist-test-slave-fjww krb5kdc[8625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564659, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:19.921262 8638 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:04:19.922916 8638 webserver.cc:460] Webserver started at http://127.6.192.62:44049/ using document root <none> and password file <none>
I20260928 03:04:19.923727 8638 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:19.923861 8638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:19.924197 8638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:19.927240 8638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "d6df20f9377841bba26fc46273e484cf"
format_stamp: "Formatted at 2026-09-28 03:04:19 on dist-test-slave-fjww"
server_key: "b50ad4079b79d1488fcc95a10e92d218"
server_key_iv: "7f354c04681f486c8b68465813d667c2"
server_key_version: "encryptionkey@0"
I20260928 03:04:19.928120 8638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "d6df20f9377841bba26fc46273e484cf"
format_stamp: "Formatted at 2026-09-28 03:04:19 on dist-test-slave-fjww"
server_key: "b50ad4079b79d1488fcc95a10e92d218"
server_key_iv: "7f354c04681f486c8b68465813d667c2"
server_key_version: "encryptionkey@0"
I20260928 03:04:19.934082 8638 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 03:04:19.938129 8653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:19.939859 8638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 03:04:19.940061 8638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "d6df20f9377841bba26fc46273e484cf"
format_stamp: "Formatted at 2026-09-28 03:04:19 on dist-test-slave-fjww"
server_key: "b50ad4079b79d1488fcc95a10e92d218"
server_key_iv: "7f354c04681f486c8b68465813d667c2"
server_key_version: "encryptionkey@0"
I20260928 03:04:19.940301 8638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:19.967293 8638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:19.972664 8638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:19.973001 8638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:19.996040 8638 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:38929
I20260928 03:04:19.996047 8714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:38929 every 8 connection(s)
I20260928 03:04:19.997622 8638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:04:20.000134 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 8638
I20260928 03:04:20.000480 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:04:20.001210 6912 external_mini_cluster.cc:1428] Setting key 9f20fe2db153fb62a5e6bf8b24b8f832
I20260928 03:04:20.002337 8715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:20.017326 8715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf: Bootstrap starting.
Sep 28 03:04:20 dist-test-slave-fjww krb5kdc[8625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:20.021263 8715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:20.022375 8715 log.cc:824] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:20.025569 8715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf: No bootstrap required, opened a new log
I20260928 03:04:20.028029 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:20.003611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33072 (local address 127.6.192.62:38929)
0928 03:04:20.004516 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:20.004539 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:20.004593 (+ 54us) server_negotiation.cc:407] Connection header received
0928 03:04:20.006377 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:20.006408 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:20.007016 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:20.007554 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:20.009058 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:20.011162 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:20.012751 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:20.013905 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:20.014139 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:20.019157 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:20.019176 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:20.019202 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:20.019299 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:20.023039 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:20.024026 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:20.024033 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:20.024041 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:20.024258 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:20.024730 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:20.024734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:20.024737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:20.025344 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:20.025695 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:20.026272 (+ 577us) server_negotiation.cc:299] Negotiation successful
0928 03:04:20.026764 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":383,"threads_started":1}
I20260928 03:04:20.034533 8715 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6df20f9377841bba26fc46273e484cf" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38929 } }
I20260928 03:04:20.034982 8715 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:20.035107 8715 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6df20f9377841bba26fc46273e484cf, State: Initialized, Role: FOLLOWER
I20260928 03:04:20.035859 8715 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6df20f9377841bba26fc46273e484cf" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38929 } }
I20260928 03:04:20.036172 8715 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:04:20.036326 8715 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:04:20.036520 8715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:20.038800 8715 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6df20f9377841bba26fc46273e484cf" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38929 } }
I20260928 03:04:20.039302 8715 leader_election.cc:302] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6df20f9377841bba26fc46273e484cf; no voters:
I20260928 03:04:20.039991 8715 leader_election.cc:288] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:04:20.040185 8720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:20.041146 8720 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf [term 1 LEADER]: Becoming Leader. State: Replica: d6df20f9377841bba26fc46273e484cf, State: Running, Role: LEADER
I20260928 03:04:20.041774 8720 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6df20f9377841bba26fc46273e484cf" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38929 } }
I20260928 03:04:20.042296 8715 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:04:20.045302 8722 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6df20f9377841bba26fc46273e484cf. Latest consensus state: current_term: 1 leader_uuid: "d6df20f9377841bba26fc46273e484cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6df20f9377841bba26fc46273e484cf" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38929 } } }
I20260928 03:04:20.045387 8721 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6df20f9377841bba26fc46273e484cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6df20f9377841bba26fc46273e484cf" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38929 } } }
I20260928 03:04:20.045768 8722 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:20.045809 8721 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:20.046814 8726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:04:20.054401 8726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:04:20.065176 8726 catalog_manager.cc:1347] Generated new cluster ID: 14865e16d1f243718ca79aa969509fa6
I20260928 03:04:20.065330 8726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:04:20.094403 8726 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:04:20.096427 8726 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:04:20.111104 8726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf: Generated new TSK 0
I20260928 03:04:20.112285 8726 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
I20260928 03:04:20.164556 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:38929
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:20.509974 8741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:20.510272 8741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:20.510389 8741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:20.518333 8741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:20.518464 8741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:20.518553 8741 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 03:04:20.518606 8741 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 03:04:20.518677 8741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:04:20.529341 8741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.192.62:38929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:20.530632 8741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:20.532095 8741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:20.546082 8746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:20.546144 8747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:20.546442 8749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:20.547535 8741 server_base.cc:1034] running on GCE node
I20260928 03:04:20.548503 8741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:20.549640 8741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:20.550930 8741 hybrid_clock.cc:648] HybridClock initialized: now 1790564660550849 us; error 74 us; skew 500 ppm
I20260928 03:04:20.551329 8741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:20 dist-test-slave-fjww krb5kdc[8625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564660, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:20.554983 8741 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:04:20.556958 8741 webserver.cc:460] Webserver started at http://127.6.192.1:45191/ using document root <none> and password file <none>
I20260928 03:04:20.557696 8741 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:20.557852 8741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:20.558195 8741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:20.561403 8741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "493349743b1a44ba839d4087a371c106"
format_stamp: "Formatted at 2026-09-28 03:04:20 on dist-test-slave-fjww"
server_key: "5cee975e5a4d21a2e2733011ab28facc"
server_key_iv: "8de309b03cee1bf275b769cfe50b8960"
server_key_version: "encryptionkey@0"
I20260928 03:04:20.562258 8741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "493349743b1a44ba839d4087a371c106"
format_stamp: "Formatted at 2026-09-28 03:04:20 on dist-test-slave-fjww"
server_key: "5cee975e5a4d21a2e2733011ab28facc"
server_key_iv: "8de309b03cee1bf275b769cfe50b8960"
server_key_version: "encryptionkey@0"
I20260928 03:04:20.568122 8741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 03:04:20.571954 8756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:20.573374 8741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:04:20.573558 8741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "493349743b1a44ba839d4087a371c106"
format_stamp: "Formatted at 2026-09-28 03:04:20 on dist-test-slave-fjww"
server_key: "5cee975e5a4d21a2e2733011ab28facc"
server_key_iv: "8de309b03cee1bf275b769cfe50b8960"
server_key_version: "encryptionkey@0"
I20260928 03:04:20.573772 8741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:20.591271 8741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:20.595959 8741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:20.596271 8741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:20.599545 8741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:20.599684 8741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:20.599833 8741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:20.599931 8741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 03:04:20 dist-test-slave-fjww krb5kdc[8625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564660, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:20.627080 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:20.600097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:41305 (local address 127.6.192.62:38929)
0928 03:04:20.603915 (+ 3818us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:20.603922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:20.603948 (+ 26us) server_negotiation.cc:407] Connection header received
0928 03:04:20.605493 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:20.605499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:20.605600 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:20.605742 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:20.608800 (+ 3058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:20.609895 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:20.613233 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:20.613668 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:20.613792 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:20.620147 (+ 6355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:20.620153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:20.620157 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:20.620196 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:20.623678 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:20.624642 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:20.624647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:20.624649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:20.624783 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:20.625266 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:20.625271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:20.625276 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:20.625561 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:20.625728 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:20.626476 (+ 748us) server_negotiation.cc:299] Negotiation successful
0928 03:04:20.626700 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3652,"thread_start_us":3513,"threads_started":1}
I20260928 03:04:20.628865 8766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:20.600790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:38929 (local address 127.6.192.1:41305)
0928 03:04:20.601983 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:20.602066 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:20.605054 (+ 2988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:20.606103 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:20.606124 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:20.607304 (+ 1180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:20.608511 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:20.608549 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:20.610097 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:20.610106 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:20.613018 (+ 2912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:20.613046 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:20.613969 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:20.613980 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:20.614386 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:20.615375 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:20.615420 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:20.619724 (+ 4304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:20.623863 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:20.623875 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:20.623894 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:20.624489 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:20.624901 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:20.624907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:20.624910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:20.625135 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:20.625696 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:20.625704 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:20.626093 (+ 389us) client_negotiation.cc:769] Sending connection context
0928 03:04:20.626384 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 03:04:20.626962 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":447,"threads_started":1}
I20260928 03:04:20.652176 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:20.636291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:35597 (local address 127.6.192.62:38929)
0928 03:04:20.636536 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:20.636545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:20.637246 (+ 701us) server_negotiation.cc:407] Connection header received
0928 03:04:20.637493 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:20.637499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:20.637579 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:20.637686 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:20.639249 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:20.640253 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:20.641580 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:20.642037 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:20.642202 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:20.645322 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:20.645328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:20.645331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:20.645367 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:20.648458 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:20.649688 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:20.649697 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:20.649702 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:20.649890 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:20.650492 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:20.650500 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:20.650506 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:20.650907 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:20.651091 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:20.651558 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 03:04:20.651817 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 03:04:20.652350 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:20.636373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:38929 (local address 127.6.192.1:35597)
0928 03:04:20.637108 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:20.637137 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:20.637336 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:20.637710 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:20.637714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:20.638412 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:20.639100 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:20.639112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:20.640396 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:20.640413 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:20.641436 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:20.641447 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:20.642289 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:20.642299 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:20.642587 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:20.643503 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:20.643537 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:20.645046 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:20.648683 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:20.648688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:20.648692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:20.649516 (+ 824us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:20.650008 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:20.650014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:20.650018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:20.650326 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:20.651051 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:20.651056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:20.651259 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 03:04:20.651413 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 03:04:20.651869 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":418,"threads_started":1}
I20260928 03:04:20.661510 8741 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:37379
I20260928 03:04:20.661523 8876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:37379 every 8 connection(s)
I20260928 03:04:20.662883 8741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:04:20.665473 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 8741
I20260928 03:04:20.665552 8877 heartbeater.cc:347] Connected to a master server at 127.6.192.62:38929
I20260928 03:04:20.665732 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:04:20.665802 8877 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:20.666606 6912 external_mini_cluster.cc:1428] Setting key 76c4bd7470670b88c8591a3b8102d0e6
I20260928 03:04:20.666785 8877 heartbeater.cc:511] Master 127.6.192.62:38929 requested a full tablet report, sending...
I20260928 03:04:20.669205 8669 ts_manager.cc:194] Registered new tserver with Master: 493349743b1a44ba839d4087a371c106 (127.6.192.1:37379)
I20260928 03:04:20.671518 8669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:41305
WARNING: no policy specified for kudu/127.6.192.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
I20260928 03:04:20.714943 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:38929
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:21.061421 8881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:21.061729 8881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:21.061833 8881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:21.070698 8881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:21.070861 8881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:21.070974 8881 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 03:04:21.071085 8881 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 03:04:21.071182 8881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:04:21.082127 8881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.192.62:38929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:21.083670 8881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:21.085110 8881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:21.099921 8886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:21.100023 8887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:21.099942 8889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:21.101538 8881 server_base.cc:1034] running on GCE node
I20260928 03:04:21.102301 8881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:21.103386 8881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:21.104671 8881 hybrid_clock.cc:648] HybridClock initialized: now 1790564661104601 us; error 83 us; skew 500 ppm
I20260928 03:04:21.105028 8881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:21 dist-test-slave-fjww krb5kdc[8625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564661, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:21.108793 8881 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:04:21.110590 8881 webserver.cc:460] Webserver started at http://127.6.192.2:44005/ using document root <none> and password file <none>
I20260928 03:04:21.111398 8881 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:21.111630 8881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:21.111989 8881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:21.115377 8881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "d6eb96db463547beb7a35b57f6bf6fab"
format_stamp: "Formatted at 2026-09-28 03:04:21 on dist-test-slave-fjww"
server_key: "1b12c87ceb6c8a09536d8bdbe16649e9"
server_key_iv: "f97c2e8d9880defb9e5c1a3306e47a61"
server_key_version: "encryptionkey@0"
I20260928 03:04:21.116386 8881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "d6eb96db463547beb7a35b57f6bf6fab"
format_stamp: "Formatted at 2026-09-28 03:04:21 on dist-test-slave-fjww"
server_key: "1b12c87ceb6c8a09536d8bdbe16649e9"
server_key_iv: "f97c2e8d9880defb9e5c1a3306e47a61"
server_key_version: "encryptionkey@0"
I20260928 03:04:21.122571 8881 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 03:04:21.127321 8896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:21.129068 8881 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 03:04:21.129320 8881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "d6eb96db463547beb7a35b57f6bf6fab"
format_stamp: "Formatted at 2026-09-28 03:04:21 on dist-test-slave-fjww"
server_key: "1b12c87ceb6c8a09536d8bdbe16649e9"
server_key_iv: "f97c2e8d9880defb9e5c1a3306e47a61"
server_key_version: "encryptionkey@0"
I20260928 03:04:21.129635 8881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:21.142328 8881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:21.147013 8881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:21.147310 8881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:21.150213 8881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:21.150323 8881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:21.150458 8881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:21.150558 8881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 03:04:21 dist-test-slave-fjww krb5kdc[8625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564661, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:21.175019 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:21.150672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:46017 (local address 127.6.192.62:38929)
0928 03:04:21.150887 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:21.150893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:21.152591 (+ 1698us) server_negotiation.cc:407] Connection header received
0928 03:04:21.154734 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:21.154742 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:21.154881 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:21.155037 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:21.157457 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.158284 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.161171 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.161936 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.162108 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:21.166860 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:21.166867 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:21.166873 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:21.166934 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:21.170971 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:21.172166 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:21.172172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:21.172176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:21.172319 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:21.172991 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:21.172997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:21.173003 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:21.173430 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:21.173606 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:21.174382 (+ 776us) server_negotiation.cc:299] Negotiation successful
0928 03:04:21.174648 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 03:04:21.176213 8905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:21.151429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:38929 (local address 127.6.192.2:46017)
0928 03:04:21.152389 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:21.152447 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:21.154419 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:21.155326 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:21.155348 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:21.156186 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:21.157223 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:21.157266 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.158451 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.158459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:21.160955 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:21.160984 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.162265 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.162277 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:21.162694 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:21.163683 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:21.163711 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:21.166544 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:21.171215 (+ 4671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:21.171223 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:21.171241 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:21.172013 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:21.172451 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:21.172458 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:21.172462 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:21.172852 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:21.173570 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:21.173579 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:21.174004 (+ 425us) client_negotiation.cc:769] Sending connection context
0928 03:04:21.174287 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 03:04:21.174880 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":405,"threads_started":1}
I20260928 03:04:21.198781 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:21.183414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:40459 (local address 127.6.192.62:38929)
0928 03:04:21.183699 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:21.183709 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:21.184324 (+ 615us) server_negotiation.cc:407] Connection header received
0928 03:04:21.184469 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:21.184478 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:21.184585 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:21.184753 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:21.185832 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.187332 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.189623 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.190098 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.190262 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:21.192334 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:21.192339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:21.192343 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:21.192375 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:21.196010 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:21.196746 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:21.196758 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:21.196761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:21.196908 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:21.197296 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:21.197305 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:21.197307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:21.197647 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:21.197807 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:21.198209 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 03:04:21.198439 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 03:04:21.198925 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:21.183393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:38929 (local address 127.6.192.2:40459)
0928 03:04:21.184187 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:21.184219 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:21.184353 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:21.184784 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:21.184791 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:21.185282 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:21.185667 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:21.185677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.187589 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.187597 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:21.189440 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:21.189459 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.190307 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.190313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:21.190514 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:21.191201 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:21.191219 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:21.192111 (+ 892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:21.196162 (+ 4051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:21.196167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:21.196171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:21.196606 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:21.197027 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:21.197032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:21.197036 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:21.197172 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:21.197770 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:21.197775 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:21.197962 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 03:04:21.198103 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 03:04:21.198457 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":446,"threads_started":1}
I20260928 03:04:21.211339 8881 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:46685
I20260928 03:04:21.211383 9015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:46685 every 8 connection(s)
I20260928 03:04:21.212713 8881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:04:21.214954 9016 heartbeater.cc:347] Connected to a master server at 127.6.192.62:38929
I20260928 03:04:21.215188 9016 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:21.216007 9016 heartbeater.cc:511] Master 127.6.192.62:38929 requested a full tablet report, sending...
I20260928 03:04:21.216269 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 8881
I20260928 03:04:21.216459 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:04:21.217041 6912 external_mini_cluster.cc:1428] Setting key 3138e256c146a0237947a1f1cb4c63c3
I20260928 03:04:21.217821 8669 ts_manager.cc:194] Registered new tserver with Master: d6eb96db463547beb7a35b57f6bf6fab (127.6.192.2:46685)
I20260928 03:04:21.218801 8669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:46017
WARNING: no policy specified for kudu/127.6.192.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
I20260928 03:04:21.263892 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:38929
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:21.600931 9020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:21.601352 9020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:21.601541 9020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:21.609078 9020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:21.609256 9020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:21.609426 9020 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 03:04:21.609519 9020 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 03:04:21.609633 9020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:04:21.620297 9020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.192.62:38929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:21.621869 9020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:21.623334 9020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:21.638157 9026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:21.638170 9028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:21.638759 9025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:21.638981 9020 server_base.cc:1034] running on GCE node
I20260928 03:04:21.639724 9020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:21.640651 9020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:21.641896 9020 hybrid_clock.cc:648] HybridClock initialized: now 1790564661641828 us; error 42 us; skew 500 ppm
I20260928 03:04:21.642217 9020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:21 dist-test-slave-fjww krb5kdc[8625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564661, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:21.645839 9020 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:04:21.647518 9020 webserver.cc:460] Webserver started at http://127.6.192.3:45641/ using document root <none> and password file <none>
I20260928 03:04:21.648317 9020 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:21.648456 9020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:21.648782 9020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:21.651852 9020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "e5489b108dc84563bb1541d4a52e1a49"
format_stamp: "Formatted at 2026-09-28 03:04:21 on dist-test-slave-fjww"
server_key: "10d3741eca05e2c957331e1329588567"
server_key_iv: "ba3ff15f4bf7125aa48a115de599eec1"
server_key_version: "encryptionkey@0"
I20260928 03:04:21.652686 9020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "e5489b108dc84563bb1541d4a52e1a49"
format_stamp: "Formatted at 2026-09-28 03:04:21 on dist-test-slave-fjww"
server_key: "10d3741eca05e2c957331e1329588567"
server_key_iv: "ba3ff15f4bf7125aa48a115de599eec1"
server_key_version: "encryptionkey@0"
I20260928 03:04:21.658185 9020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:04:21.661872 9035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:21.663632 9020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:04:21.663805 9020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "e5489b108dc84563bb1541d4a52e1a49"
format_stamp: "Formatted at 2026-09-28 03:04:21 on dist-test-slave-fjww"
server_key: "10d3741eca05e2c957331e1329588567"
server_key_iv: "ba3ff15f4bf7125aa48a115de599eec1"
server_key_version: "encryptionkey@0"
I20260928 03:04:21.664055 9020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:21.675297 8877 heartbeater.cc:503] Master 127.6.192.62:38929 was elected leader, sending a full tablet report...
I20260928 03:04:21.679128 9020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:21.683339 9020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:21.683661 9020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:21.686338 9020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:21.686435 9020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:21.686595 9020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:21.686686 9020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 03:04:21 dist-test-slave-fjww krb5kdc[8625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564661, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:21.712826 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:21.687256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:58443 (local address 127.6.192.62:38929)
0928 03:04:21.687480 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:21.687490 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:21.689084 (+ 1594us) server_negotiation.cc:407] Connection header received
0928 03:04:21.692355 (+ 3271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:21.692361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:21.692470 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:21.692595 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:21.695196 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.696312 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.698898 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.699763 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.699936 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:21.704653 (+ 4717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:21.704660 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:21.704664 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:21.704708 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:21.708942 (+ 4234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:21.710013 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:21.710018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:21.710022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:21.710207 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:21.710820 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:21.710826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:21.710829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:21.711283 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:21.711475 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:21.712208 (+ 733us) server_negotiation.cc:299] Negotiation successful
0928 03:04:21.712467 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 03:04:21.714236 9044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:21.687935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:38929 (local address 127.6.192.3:58443)
0928 03:04:21.688876 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:21.688932 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:21.691922 (+ 2990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:21.692937 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:21.692946 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:21.693880 (+ 934us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:21.694898 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:21.694934 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.696492 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.696504 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:21.698698 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:21.698723 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.700105 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.700111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:21.700302 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:21.701042 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:21.701065 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:21.704252 (+ 3187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:21.709130 (+ 4878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:21.709147 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:21.709169 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:21.709866 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:21.710367 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:21.710373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:21.710382 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:21.710660 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:21.711427 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:21.711435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:21.711829 (+ 394us) client_negotiation.cc:769] Sending connection context
0928 03:04:21.712115 (+ 286us) client_negotiation.cc:240] Negotiation successful
0928 03:04:21.712697 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":351,"threads_started":1}
I20260928 03:04:21.737504 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:21.721280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:48927 (local address 127.6.192.62:38929)
0928 03:04:21.721458 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:21.721462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:21.722098 (+ 636us) server_negotiation.cc:407] Connection header received
0928 03:04:21.722304 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:21.722308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:21.722373 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:21.722478 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:21.723531 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.724518 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.727246 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.727951 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.728095 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:21.730189 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:21.730198 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:21.730206 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:21.730275 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:21.734449 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:21.735177 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:21.735182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:21.735193 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:21.735320 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:21.735735 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:21.735741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:21.735747 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:21.736171 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:21.736343 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:21.736913 (+ 570us) server_negotiation.cc:299] Negotiation successful
0928 03:04:21.737152 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 03:04:21.737761 9119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:21.721339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:38929 (local address 127.6.192.3:48927)
0928 03:04:21.721988 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:21.722004 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:21.722166 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:21.722502 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:21.722506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:21.722995 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:21.723308 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:21.723317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.724702 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.724710 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:21.727025 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:21.727045 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.728134 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.728139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:21.728317 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:21.729133 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:21.729151 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:21.729944 (+ 793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:21.734622 (+ 4678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:21.734629 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:21.734633 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:21.735045 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:21.735468 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:21.735474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:21.735478 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:21.735622 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:21.736312 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:21.736316 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:21.736599 (+ 283us) client_negotiation.cc:769] Sending connection context
0928 03:04:21.736814 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 03:04:21.737189 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":378,"threads_started":1}
I20260928 03:04:21.745700 9020 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:46441
I20260928 03:04:21.745729 9154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:46441 every 8 connection(s)
I20260928 03:04:21.747047 9020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:04:21.749351 9155 heartbeater.cc:347] Connected to a master server at 127.6.192.62:38929
I20260928 03:04:21.749677 9155 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:21.750393 9155 heartbeater.cc:511] Master 127.6.192.62:38929 requested a full tablet report, sending...
I20260928 03:04:21.751933 8669 ts_manager.cc:194] Registered new tserver with Master: e5489b108dc84563bb1541d4a52e1a49 (127.6.192.3:46441)
I20260928 03:04:21.752815 8669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:58443
I20260928 03:04:21.754160 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 9020
I20260928 03:04:21.754328 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:04:21.754787 6912 external_mini_cluster.cc:1428] Setting key 3af95e34e02fc8e37d1934390372af4d
I20260928 03:04:21.757804 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 03:04:21 dist-test-slave-fjww krb5kdc[8625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 03:04:21 dist-test-slave-fjww krb5kdc[8625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:21.806564 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:21.792642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49542 (local address 127.6.192.62:38929)
0928 03:04:21.792906 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:21.792914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:21.793226 (+ 312us) server_negotiation.cc:407] Connection header received
0928 03:04:21.793362 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:21.793366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:21.793432 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:21.793521 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:21.794885 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.795710 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.796892 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.797297 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.797396 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:21.801353 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:21.801358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:21.801361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:21.801402 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:21.804139 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:21.804829 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:21.804833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:21.804835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:21.804968 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:21.805302 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:21.805305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:21.805307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:21.805573 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:21.805762 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:21.806074 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 03:04:21.806248 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 03:04:21.811219 8669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:04:21.818542 8669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:04:21.846107 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:21.832438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:37379 (local address 127.0.0.1:50162)
0928 03:04:21.834184 (+ 1746us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:21.834220 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:21.834491 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:21.835653 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:21.835665 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:21.835708 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:21.836208 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:21.836225 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.838599 (+ 2374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.838608 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:21.841624 (+ 3016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:21.841649 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.844692 (+ 3043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.844703 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:21.845050 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:21.845131 (+ 81us) client_negotiation.cc:769] Sending connection context
0928 03:04:21.845293 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 03:04:21.845584 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1488,"thread_start_us":1344,"threads_started":1}
I20260928 03:04:21.847131 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:21.832715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50162 (local address 127.6.192.1:37379)
0928 03:04:21.833585 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:21.833598 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:21.834326 (+ 728us) server_negotiation.cc:407] Connection header received
0928 03:04:21.834750 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:21.834761 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:21.835087 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:21.835267 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:21.836426 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.838340 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.841874 (+ 3534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.843986 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.844134 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:21.844449 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:21.846572 (+ 2123us) server_negotiation.cc:299] Negotiation successful
0928 03:04:21.846769 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":486,"threads_started":1}
I20260928 03:04:21.851732 8807 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc6a4d322c74202bc01ba02ef9606d3 (DEFAULT_TABLE table=test-table [id=ffdd871f2e07411083d7bfac9d6b2401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:21.853387 8807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc6a4d322c74202bc01ba02ef9606d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:21.855495 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:21.834615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:46685 (local address 127.0.0.1:55346)
0928 03:04:21.847349 (+ 12734us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:21.847370 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:21.847518 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:21.848494 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:21.848499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:21.848526 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:21.848895 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:21.848915 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.850544 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.850550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:21.852700 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:21.852716 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.854581 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.854588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:21.854770 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:21.854808 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 03:04:21.854929 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 03:04:21.855094 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12612,"thread_start_us":12447,"threads_started":1}
I20260928 03:04:21.855504 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:21.835093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.6.192.2:46685)
0928 03:04:21.846908 (+ 11815us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:21.846916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:21.847644 (+ 728us) server_negotiation.cc:407] Connection header received
0928 03:04:21.847768 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:21.847777 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:21.848064 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:21.848271 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:21.849072 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.850391 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.852918 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.854361 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.854508 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:21.854716 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:21.854999 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 03:04:21.855182 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11646,"thread_start_us":11512,"threads_started":1}
I20260928 03:04:21.855816 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:21.833287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:46441 (local address 127.0.0.1:49088)
0928 03:04:21.845705 (+ 12418us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:21.845731 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:21.845898 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:21.847764 (+ 1866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:21.847769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:21.847796 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:21.848226 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:21.848240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.851274 (+ 3034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.851280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:21.853367 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:21.853379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.855059 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.855065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:21.855240 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:21.855272 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 03:04:21.855344 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 03:04:21.855511 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11319,"thread_start_us":11404,"threads_started":1}
I20260928 03:04:21.857218 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:21.833953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49088 (local address 127.6.192.3:46441)
0928 03:04:21.846735 (+ 12782us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:21.846744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:21.846780 (+ 36us) server_negotiation.cc:407] Connection header received
0928 03:04:21.846885 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:21.846895 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:21.847216 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:21.848450 (+ 1234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:21.849807 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.851124 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.853674 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:21.854839 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:21.856362 (+ 1523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:21.856619 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:21.856699 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 03:04:21.856872 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12482,"thread_start_us":12275,"threads_started":1}
I20260928 03:04:21.859664 8946 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc6a4d322c74202bc01ba02ef9606d3 (DEFAULT_TABLE table=test-table [id=ffdd871f2e07411083d7bfac9d6b2401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:21.860879 8946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc6a4d322c74202bc01ba02ef9606d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:21.860394 9085 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc6a4d322c74202bc01ba02ef9606d3 (DEFAULT_TABLE table=test-table [id=ffdd871f2e07411083d7bfac9d6b2401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:21.861800 9085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc6a4d322c74202bc01ba02ef9606d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:21.872299 9174 tablet_bootstrap.cc:492] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab: Bootstrap starting.
I20260928 03:04:21.874466 9175 tablet_bootstrap.cc:492] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106: Bootstrap starting.
I20260928 03:04:21.876941 9174 tablet_bootstrap.cc:654] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:21.877323 9176 tablet_bootstrap.cc:492] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49: Bootstrap starting.
I20260928 03:04:21.878408 9175 tablet_bootstrap.cc:654] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:21.878536 9174 log.cc:824] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:21.879817 9175 log.cc:824] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:21.881007 9176 tablet_bootstrap.cc:654] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:21.881651 9174 tablet_bootstrap.cc:492] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab: No bootstrap required, opened a new log
I20260928 03:04:21.881930 9174 ts_tablet_manager.cc:1397] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 03:04:21.882258 9176 log.cc:824] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:21.882879 9175 tablet_bootstrap.cc:492] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106: No bootstrap required, opened a new log
I20260928 03:04:21.883128 9175 ts_tablet_manager.cc:1397] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 03:04:21.886227 9176 tablet_bootstrap.cc:492] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49: No bootstrap required, opened a new log
I20260928 03:04:21.886485 9176 ts_tablet_manager.cc:1397] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 03:04:21.890236 9174 raft_consensus.cc:348] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:21.890735 9174 raft_consensus.cc:374] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:21.890843 9174 raft_consensus.cc:729] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6eb96db463547beb7a35b57f6bf6fab, State: Initialized, Role: FOLLOWER
I20260928 03:04:21.891609 9174 consensus_queue.cc:260] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:21.892694 9016 heartbeater.cc:503] Master 127.6.192.62:38929 was elected leader, sending a full tablet report...
I20260928 03:04:21.893818 9175 raft_consensus.cc:348] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:21.894279 9175 raft_consensus.cc:374] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:21.894375 9175 raft_consensus.cc:729] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493349743b1a44ba839d4087a371c106, State: Initialized, Role: FOLLOWER
I20260928 03:04:21.894740 9174 ts_tablet_manager.cc:1428] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260928 03:04:21.895126 9175 consensus_queue.cc:260] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:21.897926 9176 raft_consensus.cc:348] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:21.898406 9176 raft_consensus.cc:374] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:21.898509 9176 raft_consensus.cc:729] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5489b108dc84563bb1541d4a52e1a49, State: Initialized, Role: FOLLOWER
I20260928 03:04:21.898656 9175 ts_tablet_manager.cc:1428] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260928 03:04:21.899127 9176 consensus_queue.cc:260] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:21.900344 9155 heartbeater.cc:503] Master 127.6.192.62:38929 was elected leader, sending a full tablet report...
I20260928 03:04:21.901153 9176 ts_tablet_manager.cc:1428] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260928 03:04:21.915791 8878 tablet.cc:2369] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:04:21.964902 9017 tablet.cc:2369] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:04:21.998857 9156 tablet.cc:2369] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:04:22.147794 9180 raft_consensus.cc:482] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:04:22.148136 9180 raft_consensus.cc:504] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.150045 9180 leader_election.cc:288] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 493349743b1a44ba839d4087a371c106 (127.6.192.1:37379), e5489b108dc84563bb1541d4a52e1a49 (127.6.192.3:46441)
I20260928 03:04:22.157604 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:22.150460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:45493 (local address 127.6.192.3:46441)
0928 03:04:22.150739 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:22.150747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:22.151849 (+ 1102us) server_negotiation.cc:407] Connection header received
0928 03:04:22.152053 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:22.152057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:22.152116 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:22.152216 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:22.152951 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.153666 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.155754 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.156535 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.156687 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:22.156816 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:22.157186 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 03:04:22.157320 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260928 03:04:22.157603 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:22.150496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:37031 (local address 127.6.192.1:37379)
0928 03:04:22.150772 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:22.150780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:22.151753 (+ 973us) server_negotiation.cc:407] Connection header received
0928 03:04:22.151997 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:22.152001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:22.152072 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:22.152167 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:22.152997 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.153698 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.155737 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.156536 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.156719 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:22.156850 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:22.157215 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 03:04:22.157347 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 03:04:22.157812 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:22.150578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:46441 (local address 127.6.192.2:45493)
0928 03:04:22.151729 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:22.151744 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:22.151900 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:22.152231 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:22.152235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:22.152259 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:22.152767 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:22.152777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.153824 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.153831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:22.155590 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:22.155603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.156794 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.156803 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:22.156982 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:22.157044 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 03:04:22.157155 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 03:04:22.157305 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":81,"thread_start_us":653,"threads_started":1}
I20260928 03:04:22.158192 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:22.150478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:37379 (local address 127.6.192.2:37031)
0928 03:04:22.151619 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:22.151639 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:22.151825 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:22.152196 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:22.152200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:22.152225 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:22.152840 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:22.152850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.153827 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.153835 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:22.155544 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:22.155561 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.156832 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.156839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:22.157050 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:22.157074 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 03:04:22.157174 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 03:04:22.157310 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"mutex_wait_us":86,"thread_start_us":727,"threads_started":1}
I20260928 03:04:22.158903 9105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc6a4d322c74202bc01ba02ef9606d3" candidate_uuid: "d6eb96db463547beb7a35b57f6bf6fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5489b108dc84563bb1541d4a52e1a49" is_pre_election: true
I20260928 03:04:22.159272 8827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc6a4d322c74202bc01ba02ef9606d3" candidate_uuid: "d6eb96db463547beb7a35b57f6bf6fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493349743b1a44ba839d4087a371c106" is_pre_election: true
I20260928 03:04:22.159623 9105 raft_consensus.cc:2454] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6eb96db463547beb7a35b57f6bf6fab in term 0.
I20260928 03:04:22.159847 8827 raft_consensus.cc:2454] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6eb96db463547beb7a35b57f6bf6fab in term 0.
I20260928 03:04:22.160420 8897 leader_election.cc:302] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6eb96db463547beb7a35b57f6bf6fab, e5489b108dc84563bb1541d4a52e1a49; no voters:
I20260928 03:04:22.160845 9180 raft_consensus.cc:2781] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:04:22.160995 9180 raft_consensus.cc:482] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:04:22.161105 9180 raft_consensus.cc:3037] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:22.163261 9180 raft_consensus.cc:504] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.163964 9180 leader_election.cc:288] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [CANDIDATE]: Term 1 election: Requested vote from peers 493349743b1a44ba839d4087a371c106 (127.6.192.1:37379), e5489b108dc84563bb1541d4a52e1a49 (127.6.192.3:46441)
I20260928 03:04:22.164462 8827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc6a4d322c74202bc01ba02ef9606d3" candidate_uuid: "d6eb96db463547beb7a35b57f6bf6fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493349743b1a44ba839d4087a371c106"
I20260928 03:04:22.164500 9105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc6a4d322c74202bc01ba02ef9606d3" candidate_uuid: "d6eb96db463547beb7a35b57f6bf6fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5489b108dc84563bb1541d4a52e1a49"
I20260928 03:04:22.164705 8827 raft_consensus.cc:3037] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:22.164711 9105 raft_consensus.cc:3037] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:22.166913 9105 raft_consensus.cc:2454] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6eb96db463547beb7a35b57f6bf6fab in term 1.
I20260928 03:04:22.167078 8827 raft_consensus.cc:2454] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6eb96db463547beb7a35b57f6bf6fab in term 1.
I20260928 03:04:22.167353 8897 leader_election.cc:302] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6eb96db463547beb7a35b57f6bf6fab, e5489b108dc84563bb1541d4a52e1a49; no voters:
I20260928 03:04:22.167706 9180 raft_consensus.cc:2781] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:22.168160 9180 raft_consensus.cc:686] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [term 1 LEADER]: Becoming Leader. State: Replica: d6eb96db463547beb7a35b57f6bf6fab, State: Running, Role: LEADER
I20260928 03:04:22.168749 9180 consensus_queue.cc:237] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.173839 8669 catalog_manager.cc:5495] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab reported cstate change: term changed from 0 to 1, leader changed from <none> to d6eb96db463547beb7a35b57f6bf6fab (127.6.192.2). New cstate: current_term: 1 leader_uuid: "d6eb96db463547beb7a35b57f6bf6fab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } health_report { overall_health: UNKNOWN } } }
Sep 28 03:04:22 dist-test-slave-fjww krb5kdc[8625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564659, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:22.255908 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:22.241930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49552 (local address 127.6.192.62:38929)
0928 03:04:22.242124 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:22.242132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:22.242683 (+ 551us) server_negotiation.cc:407] Connection header received
0928 03:04:22.242884 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:22.242889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:22.242951 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:22.243052 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:22.244445 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.245181 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.246444 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.246889 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.247030 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:22.250551 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:22.250555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:22.250559 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:22.250586 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:22.253163 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:22.253861 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:22.253868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:22.253874 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:22.254007 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:22.254377 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:22.254383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:22.254387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:22.254647 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:22.254796 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:22.255207 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 03:04:22.255592 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":132,"server-negotiator.queue_time_us":60}
I20260928 03:04:22.255942 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:22.241877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:38929 (local address 127.0.0.1:49552)
0928 03:04:22.242576 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:22.242591 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:22.242764 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:22.243074 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:22.243079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:22.243988 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:22.244283 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:22.244293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.245310 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.245318 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:22.246292 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:22.246304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.247096 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.247102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:22.247272 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:22.248031 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:22.248052 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:22.250348 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:22.253321 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:22.253335 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:22.253344 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:22.253743 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:22.254117 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:22.254123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:22.254127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:22.254268 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:22.254777 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:22.254785 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:22.254956 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 03:04:22.255116 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 03:04:22.255462 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":428,"threads_started":1}
I20260928 03:04:22.261427 8669 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:49552:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 03:04:22.262512 8669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:04:22.269351 8807 tablet_service.cc:1467] Processing CreateTablet for tablet f690153060344014975fb6b06867aa25 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=22de49aa956848ac8c4248880b9fc324]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 03:04:22.269651 9085 tablet_service.cc:1467] Processing CreateTablet for tablet f690153060344014975fb6b06867aa25 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=22de49aa956848ac8c4248880b9fc324]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 03:04:22.269418 8946 tablet_service.cc:1467] Processing CreateTablet for tablet f690153060344014975fb6b06867aa25 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=22de49aa956848ac8c4248880b9fc324]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 03:04:22.270404 8807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f690153060344014975fb6b06867aa25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:22.270402 9085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f690153060344014975fb6b06867aa25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:22.270414 8946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f690153060344014975fb6b06867aa25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:22.276387 9175 tablet_bootstrap.cc:492] T f690153060344014975fb6b06867aa25 P 493349743b1a44ba839d4087a371c106: Bootstrap starting.
I20260928 03:04:22.277679 9176 tablet_bootstrap.cc:492] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49: Bootstrap starting.
I20260928 03:04:22.277992 9175 tablet_bootstrap.cc:654] T f690153060344014975fb6b06867aa25 P 493349743b1a44ba839d4087a371c106: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:22.278550 9174 tablet_bootstrap.cc:492] T f690153060344014975fb6b06867aa25 P d6eb96db463547beb7a35b57f6bf6fab: Bootstrap starting.
I20260928 03:04:22.279637 9176 tablet_bootstrap.cc:654] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:22.280059 9174 tablet_bootstrap.cc:654] T f690153060344014975fb6b06867aa25 P d6eb96db463547beb7a35b57f6bf6fab: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:22.282061 9176 tablet_bootstrap.cc:492] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49: No bootstrap required, opened a new log
I20260928 03:04:22.282224 9176 ts_tablet_manager.cc:1397] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260928 03:04:22.282397 9174 tablet_bootstrap.cc:492] T f690153060344014975fb6b06867aa25 P d6eb96db463547beb7a35b57f6bf6fab: No bootstrap required, opened a new log
I20260928 03:04:22.282547 9174 ts_tablet_manager.cc:1397] T f690153060344014975fb6b06867aa25 P d6eb96db463547beb7a35b57f6bf6fab: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 03:04:22.282815 9175 tablet_bootstrap.cc:492] T f690153060344014975fb6b06867aa25 P 493349743b1a44ba839d4087a371c106: No bootstrap required, opened a new log
I20260928 03:04:22.282979 9175 ts_tablet_manager.cc:1397] T f690153060344014975fb6b06867aa25 P 493349743b1a44ba839d4087a371c106: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260928 03:04:22.283017 9176 raft_consensus.cc:348] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.283288 9176 raft_consensus.cc:374] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:22.283398 9176 raft_consensus.cc:729] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5489b108dc84563bb1541d4a52e1a49, State: Initialized, Role: FOLLOWER
I20260928 03:04:22.283252 9174 raft_consensus.cc:348] T f690153060344014975fb6b06867aa25 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.283541 9174 raft_consensus.cc:374] T f690153060344014975fb6b06867aa25 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:22.283651 9174 raft_consensus.cc:729] T f690153060344014975fb6b06867aa25 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6eb96db463547beb7a35b57f6bf6fab, State: Initialized, Role: FOLLOWER
I20260928 03:04:22.283649 9176 consensus_queue.cc:260] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.283787 9175 raft_consensus.cc:348] T f690153060344014975fb6b06867aa25 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.284027 9175 raft_consensus.cc:374] T f690153060344014975fb6b06867aa25 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:22.284127 9175 raft_consensus.cc:729] T f690153060344014975fb6b06867aa25 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493349743b1a44ba839d4087a371c106, State: Initialized, Role: FOLLOWER
I20260928 03:04:22.284319 9176 ts_tablet_manager.cc:1428] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 03:04:22.284149 9182 tablet_replica.cc:439] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } } }
I20260928 03:04:22.284345 9175 consensus_queue.cc:260] T f690153060344014975fb6b06867aa25 P 493349743b1a44ba839d4087a371c106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.284494 9182 tablet_replica.cc:442] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:04:22.283908 9174 consensus_queue.cc:260] T f690153060344014975fb6b06867aa25 P d6eb96db463547beb7a35b57f6bf6fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.284977 9175 ts_tablet_manager.cc:1428] T f690153060344014975fb6b06867aa25 P 493349743b1a44ba839d4087a371c106: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260928 03:04:22.284823 9181 tablet_replica.cc:439] T f690153060344014975fb6b06867aa25 P 493349743b1a44ba839d4087a371c106: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } } }
I20260928 03:04:22.285220 9181 tablet_replica.cc:442] T f690153060344014975fb6b06867aa25 P 493349743b1a44ba839d4087a371c106: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:04:22.285194 9180 tablet_replica.cc:439] T f690153060344014975fb6b06867aa25 P d6eb96db463547beb7a35b57f6bf6fab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } } }
I20260928 03:04:22.285493 9180 tablet_replica.cc:442] T f690153060344014975fb6b06867aa25 P d6eb96db463547beb7a35b57f6bf6fab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:04:22.286449 9174 ts_tablet_manager.cc:1428] T f690153060344014975fb6b06867aa25 P d6eb96db463547beb7a35b57f6bf6fab: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 03:04:22.300972 9182 raft_consensus.cc:482] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:04:22.301206 9182 raft_consensus.cc:504] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.302663 9182 leader_election.cc:288] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 493349743b1a44ba839d4087a371c106 (127.6.192.1:37379), d6eb96db463547beb7a35b57f6bf6fab (127.6.192.2:46685)
I20260928 03:04:22.309850 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:22.303150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:33089 (local address 127.6.192.1:37379)
0928 03:04:22.303466 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:22.303473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:22.303760 (+ 287us) server_negotiation.cc:407] Connection header received
0928 03:04:22.303979 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:22.303988 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:22.304122 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:22.304223 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:22.304731 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.305796 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.307824 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.309007 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.309103 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:22.309182 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:22.309536 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 03:04:22.309638 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 03:04:22.310052 9196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:22.303036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:37379 (local address 127.6.192.3:33089)
0928 03:04:22.303621 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:22.303656 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:22.303808 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:22.304276 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:22.304281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:22.304308 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:22.304578 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:22.304588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.305932 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.305937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:22.307663 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:22.307676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.309169 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.309175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:22.309360 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:22.309406 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 03:04:22.309505 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 03:04:22.309650 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":17,"thread_start_us":299,"threads_started":1}
I20260928 03:04:22.310813 8827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f690153060344014975fb6b06867aa25" candidate_uuid: "e5489b108dc84563bb1541d4a52e1a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493349743b1a44ba839d4087a371c106" is_pre_election: true
I20260928 03:04:22.311132 8827 raft_consensus.cc:2454] T f690153060344014975fb6b06867aa25 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5489b108dc84563bb1541d4a52e1a49 in term 0.
I20260928 03:04:22.311204 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:22.303895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:50739 (local address 127.6.192.2:46685)
0928 03:04:22.304102 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:22.304111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:22.304547 (+ 436us) server_negotiation.cc:407] Connection header received
0928 03:04:22.304733 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:22.304737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:22.304823 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:22.304907 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:22.305508 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.306582 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.308792 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.309958 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.310108 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:22.310217 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:22.310792 (+ 575us) server_negotiation.cc:299] Negotiation successful
0928 03:04:22.310931 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 03:04:22.311267 9197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:22.303751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:46685 (local address 127.6.192.3:50739)
0928 03:04:22.304432 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:22.304449 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:22.304579 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:22.304969 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:22.304973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:22.304998 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:22.305332 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:22.305343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.306741 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.306747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:22.308616 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:22.308633 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.310393 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.310436 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:22.310613 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:22.310649 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 03:04:22.310767 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 03:04:22.310899 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":4,"thread_start_us":322,"threads_started":1}
I20260928 03:04:22.311949 9039 leader_election.cc:302] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493349743b1a44ba839d4087a371c106, e5489b108dc84563bb1541d4a52e1a49; no voters:
I20260928 03:04:22.312440 9182 raft_consensus.cc:2781] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:04:22.312574 9182 raft_consensus.cc:482] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:04:22.312376 8971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f690153060344014975fb6b06867aa25" candidate_uuid: "e5489b108dc84563bb1541d4a52e1a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6eb96db463547beb7a35b57f6bf6fab" is_pre_election: true
I20260928 03:04:22.312674 9182 raft_consensus.cc:3037] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:22.312839 8971 raft_consensus.cc:2454] T f690153060344014975fb6b06867aa25 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5489b108dc84563bb1541d4a52e1a49 in term 0.
I20260928 03:04:22.315253 9182 raft_consensus.cc:504] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.315863 9182 leader_election.cc:288] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 [CANDIDATE]: Term 1 election: Requested vote from peers 493349743b1a44ba839d4087a371c106 (127.6.192.1:37379), d6eb96db463547beb7a35b57f6bf6fab (127.6.192.2:46685)
I20260928 03:04:22.316496 8827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f690153060344014975fb6b06867aa25" candidate_uuid: "e5489b108dc84563bb1541d4a52e1a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493349743b1a44ba839d4087a371c106"
I20260928 03:04:22.316807 8827 raft_consensus.cc:3037] T f690153060344014975fb6b06867aa25 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:22.316736 8971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f690153060344014975fb6b06867aa25" candidate_uuid: "e5489b108dc84563bb1541d4a52e1a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6eb96db463547beb7a35b57f6bf6fab"
I20260928 03:04:22.316999 8971 raft_consensus.cc:3037] T f690153060344014975fb6b06867aa25 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:22.319171 8827 raft_consensus.cc:2454] T f690153060344014975fb6b06867aa25 P 493349743b1a44ba839d4087a371c106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5489b108dc84563bb1541d4a52e1a49 in term 1.
I20260928 03:04:22.319425 8971 raft_consensus.cc:2454] T f690153060344014975fb6b06867aa25 P d6eb96db463547beb7a35b57f6bf6fab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5489b108dc84563bb1541d4a52e1a49 in term 1.
I20260928 03:04:22.319759 9039 leader_election.cc:302] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493349743b1a44ba839d4087a371c106, e5489b108dc84563bb1541d4a52e1a49; no voters:
I20260928 03:04:22.320118 9182 raft_consensus.cc:2781] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:22.320662 9182 raft_consensus.cc:686] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 [term 1 LEADER]: Becoming Leader. State: Replica: e5489b108dc84563bb1541d4a52e1a49, State: Running, Role: LEADER
I20260928 03:04:22.321161 9182 consensus_queue.cc:237] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.324720 9198 tablet_replica.cc:439] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49: TxnStatusTablet state changed. Reason: New leader e5489b108dc84563bb1541d4a52e1a49. Latest consensus state: current_term: 1 leader_uuid: "e5489b108dc84563bb1541d4a52e1a49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } } }
I20260928 03:04:22.325649 9198 tablet_replica.cc:442] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49: This TxnStatusTablet replica's current role is: LEADER
I20260928 03:04:22.326325 9200 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 03:04:22.326591 9200 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 03:04:22.326349 8670 catalog_manager.cc:5495] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5489b108dc84563bb1541d4a52e1a49 (127.6.192.3). New cstate: current_term: 1 leader_uuid: "e5489b108dc84563bb1541d4a52e1a49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } health_report { overall_health: HEALTHY } } }
I20260928 03:04:22.356860 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:22.349057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:46441 (local address 127.0.0.1:49098)
0928 03:04:22.349360 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:22.349385 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:22.349565 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:22.350319 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:22.350323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:22.350334 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 03:04:22.350496 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:22.350503 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.352259 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.352263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:22.353308 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:22.353323 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.354545 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.354555 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:22.354808 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:22.354916 (+ 108us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 03:04:22.356300 (+ 1384us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 03:04:22.356328 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 03:04:22.356444 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 03:04:22.356621 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 03:04:22.356954 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:22.349283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49098 (local address 127.6.192.3:46441)
0928 03:04:22.349471 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:22.349476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:22.349496 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:04:22.349836 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:22.349843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:22.349980 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:22.350135 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:04:22.350661 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.352069 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.353493 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.354295 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.354450 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:22.355300 (+ 850us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:04:22.356154 (+ 854us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:04:22.356338 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:22.356475 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 03:04:22.356601 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 03:04:22.365684 8971 raft_consensus.cc:1261] T f690153060344014975fb6b06867aa25 P d6eb96db463547beb7a35b57f6bf6fab [term 1 FOLLOWER]: Refusing update from remote peer e5489b108dc84563bb1541d4a52e1a49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:22.365703 8827 raft_consensus.cc:1261] T f690153060344014975fb6b06867aa25 P 493349743b1a44ba839d4087a371c106 [term 1 FOLLOWER]: Refusing update from remote peer e5489b108dc84563bb1541d4a52e1a49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:22.366799 9198 consensus_queue.cc:1035] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:04:22.367272 9182 consensus_queue.cc:1035] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:04:22.379595 9181 tablet_replica.cc:439] T f690153060344014975fb6b06867aa25 P 493349743b1a44ba839d4087a371c106: TxnStatusTablet state changed. Reason: New leader e5489b108dc84563bb1541d4a52e1a49. Latest consensus state: current_term: 1 leader_uuid: "e5489b108dc84563bb1541d4a52e1a49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } } }
I20260928 03:04:22.380038 9181 tablet_replica.cc:442] T f690153060344014975fb6b06867aa25 P 493349743b1a44ba839d4087a371c106: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:04:22.382596 9180 tablet_replica.cc:439] T f690153060344014975fb6b06867aa25 P d6eb96db463547beb7a35b57f6bf6fab: TxnStatusTablet state changed. Reason: New leader e5489b108dc84563bb1541d4a52e1a49. Latest consensus state: current_term: 1 leader_uuid: "e5489b108dc84563bb1541d4a52e1a49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } } }
I20260928 03:04:22.382965 9180 tablet_replica.cc:442] T f690153060344014975fb6b06867aa25 P d6eb96db463547beb7a35b57f6bf6fab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:04:22.383564 9203 mvcc.cc:205] Tried to move back new op lower bound from 7334152857038225408 to 7334152856871718912. Current Snapshot: MvccSnapshot[applied={T|T < 7334152857038225408}]
I20260928 03:04:22.387944 9205 mvcc.cc:205] Tried to move back new op lower bound from 7334152857038225408 to 7334152856871718912. Current Snapshot: MvccSnapshot[applied={T|T < 7334152857038225408}]
I20260928 03:04:22.388316 9198 tablet_replica.cc:439] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e5489b108dc84563bb1541d4a52e1a49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } } }
I20260928 03:04:22.388685 9198 tablet_replica.cc:442] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49: This TxnStatusTablet replica's current role is: LEADER
I20260928 03:04:22.395342 9180 tablet_replica.cc:439] T f690153060344014975fb6b06867aa25 P d6eb96db463547beb7a35b57f6bf6fab: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e5489b108dc84563bb1541d4a52e1a49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } } }
I20260928 03:04:22.395696 9180 tablet_replica.cc:442] T f690153060344014975fb6b06867aa25 P d6eb96db463547beb7a35b57f6bf6fab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:04:22.397930 9182 tablet_replica.cc:439] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e5489b108dc84563bb1541d4a52e1a49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } } }
I20260928 03:04:22.398240 9182 tablet_replica.cc:442] T f690153060344014975fb6b06867aa25 P e5489b108dc84563bb1541d4a52e1a49: This TxnStatusTablet replica's current role is: LEADER
I20260928 03:04:22.400799 9181 tablet_replica.cc:439] T f690153060344014975fb6b06867aa25 P 493349743b1a44ba839d4087a371c106: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e5489b108dc84563bb1541d4a52e1a49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } } }
I20260928 03:04:22.401057 9181 tablet_replica.cc:442] T f690153060344014975fb6b06867aa25 P 493349743b1a44ba839d4087a371c106: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:04:22.410818 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:22.403866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55360 (local address 127.6.192.2:46685)
0928 03:04:22.404062 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:22.404069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:22.404571 (+ 502us) server_negotiation.cc:407] Connection header received
0928 03:04:22.404750 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:22.404758 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:22.404876 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:22.405046 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:04:22.405444 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.406686 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.408151 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.408875 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.409008 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:22.409373 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:04:22.410113 (+ 740us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:04:22.410300 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:22.410355 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 03:04:22.410457 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 03:04:22.428546 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:22.421611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:46441 (local address 127.6.192.2:38939)
0928 03:04:22.422318 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:22.422341 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:22.422517 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:22.422950 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:22.422954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:22.422966 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 03:04:22.423266 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:22.423277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.424872 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.424876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:22.426150 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:22.426166 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.427300 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.427305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:22.427427 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:22.427555 (+ 128us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 03:04:22.428043 (+ 488us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 03:04:22.428051 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 03:04:22.428115 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 03:04:22.428231 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":390,"threads_started":1}
I20260928 03:04:22.428627 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:22.421658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:38939 (local address 127.6.192.3:46441)
0928 03:04:22.421886 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:22.421894 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:22.422437 (+ 543us) server_negotiation.cc:407] Connection header received
0928 03:04:22.422661 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:22.422668 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:22.422767 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:22.422894 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:04:22.423464 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.424689 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.426321 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.427116 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.427286 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:22.427771 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:04:22.427935 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:04:22.428036 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:22.428169 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 03:04:22.428275 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 03:04:22.439952 8826 raft_consensus.cc:1261] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106 [term 1 FOLLOWER]: Refusing update from remote peer d6eb96db463547beb7a35b57f6bf6fab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:22.440310 9105 raft_consensus.cc:1261] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49 [term 1 FOLLOWER]: Refusing update from remote peer d6eb96db463547beb7a35b57f6bf6fab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:22.440722 9185 consensus_queue.cc:1035] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [LEADER]: Connected to new peer: Peer: permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:04:22.441141 9180 consensus_queue.cc:1035] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:04:22.456679 9204 mvcc.cc:205] Tried to move back new op lower bound from 7334152857346621440 to 7334152856247406592. Current Snapshot: MvccSnapshot[applied={T|T < 7334152857346621440}]
I20260928 03:04:22.487311 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:22.479106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:46685 (local address 127.6.192.3:36049)
0928 03:04:22.480337 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:22.480367 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:22.480547 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:22.481127 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:22.481130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:22.481140 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 03:04:22.481585 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:22.481596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.482725 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.482731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:22.484564 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:22.484588 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.485891 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.485899 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:22.486121 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:22.486215 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 03:04:22.486754 (+ 539us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 03:04:22.486772 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 03:04:22.486854 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 03:04:22.486988 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"thread_start_us":917,"threads_started":1}
I20260928 03:04:22.487326 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:22.479736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:36049 (local address 127.6.192.2:46685)
0928 03:04:22.479970 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:22.479978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:22.480449 (+ 471us) server_negotiation.cc:407] Connection header received
0928 03:04:22.480744 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:22.480752 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:22.480852 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:22.480951 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:04:22.481766 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.482556 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.484749 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:22.485493 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:22.485659 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:22.486446 (+ 787us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:04:22.486610 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:04:22.486720 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:22.486906 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 03:04:22.487003 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 03:04:22.531821 8670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:04:22.532148 8670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:04:22.536841 8670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf: Sending DeleteTablet for 3 replicas of tablet 0fc6a4d322c74202bc01ba02ef9606d3
I20260928 03:04:22.538275 8807 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fc6a4d322c74202bc01ba02ef9606d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:22 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:50162
I20260928 03:04:22.538491 8946 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fc6a4d322c74202bc01ba02ef9606d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:22 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:55346
I20260928 03:04:22.539386 9085 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fc6a4d322c74202bc01ba02ef9606d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:22 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:49088
I20260928 03:04:22.542493 8670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:04:22.543355 8670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:04:22.554754 9225 tablet_replica.cc:330] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab: stopping tablet replica
I20260928 03:04:22.555635 9225 raft_consensus.cc:2229] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:04:22.556550 9225 raft_consensus.cc:2258] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:22.557678 8806 tablet_service.cc:1467] Processing CreateTablet for tablet 66cd0631e5cc4e0c9751aa0730149bab (DEFAULT_TABLE table=test-table [id=57743fba5c9d4a939650e08488d70b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:22.558225 8946 tablet_service.cc:1467] Processing CreateTablet for tablet 66cd0631e5cc4e0c9751aa0730149bab (DEFAULT_TABLE table=test-table [id=57743fba5c9d4a939650e08488d70b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:22.558580 8806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66cd0631e5cc4e0c9751aa0730149bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:22.558780 8946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66cd0631e5cc4e0c9751aa0730149bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:22.559367 9225 ts_tablet_manager.cc:1905] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:22.560827 9084 tablet_service.cc:1467] Processing CreateTablet for tablet 66cd0631e5cc4e0c9751aa0730149bab (DEFAULT_TABLE table=test-table [id=57743fba5c9d4a939650e08488d70b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:22.561605 9084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66cd0631e5cc4e0c9751aa0730149bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:22.565809 9175 tablet_bootstrap.cc:492] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106: Bootstrap starting.
I20260928 03:04:22.567873 9175 tablet_bootstrap.cc:654] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:22.570451 9225 ts_tablet_manager.cc:1918] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 03:04:22.570785 9225 log.cc:1196] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/0fc6a4d322c74202bc01ba02ef9606d3
I20260928 03:04:22.574008 9176 tablet_bootstrap.cc:492] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49: Bootstrap starting.
I20260928 03:04:22.576020 9176 tablet_bootstrap.cc:654] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:22.579979 9175 tablet_bootstrap.cc:492] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106: No bootstrap required, opened a new log
I20260928 03:04:22.580242 9175 ts_tablet_manager.cc:1397] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260928 03:04:22.581313 9175 raft_consensus.cc:348] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.581624 9175 raft_consensus.cc:374] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:22.581725 9175 raft_consensus.cc:729] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493349743b1a44ba839d4087a371c106, State: Initialized, Role: FOLLOWER
I20260928 03:04:22.582002 9175 consensus_queue.cc:260] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.584055 9225 ts_tablet_manager.cc:1939] T 0fc6a4d322c74202bc01ba02ef9606d3 P d6eb96db463547beb7a35b57f6bf6fab: Deleting consensus metadata
I20260928 03:04:22.586160 8657 catalog_manager.cc:4841] TS d6eb96db463547beb7a35b57f6bf6fab (127.6.192.2:46685): tablet 0fc6a4d322c74202bc01ba02ef9606d3 (table test-table [id=ffdd871f2e07411083d7bfac9d6b2401]) successfully deleted
I20260928 03:04:22.589676 9226 tablet_replica.cc:330] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49: stopping tablet replica
I20260928 03:04:22.590350 9226 raft_consensus.cc:2229] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:04:22.590754 9226 raft_consensus.cc:2258] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:22.591588 9224 tablet_replica.cc:330] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106: stopping tablet replica
I20260928 03:04:22.591814 9176 tablet_bootstrap.cc:492] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49: No bootstrap required, opened a new log
I20260928 03:04:22.592051 9176 ts_tablet_manager.cc:1397] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.005s
I20260928 03:04:22.592027 9224 raft_consensus.cc:2229] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:04:22.592438 9224 raft_consensus.cc:2258] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:22.592873 9175 ts_tablet_manager.cc:1428] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20260928 03:04:22.592943 9176 raft_consensus.cc:348] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.593178 9176 raft_consensus.cc:374] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:22.593271 9176 raft_consensus.cc:729] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5489b108dc84563bb1541d4a52e1a49, State: Initialized, Role: FOLLOWER
I20260928 03:04:22.593561 9176 consensus_queue.cc:260] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.594892 9224 ts_tablet_manager.cc:1905] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:22.596971 9226 ts_tablet_manager.cc:1905] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:22.597153 9176 ts_tablet_manager.cc:1428] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260928 03:04:22.601218 9174 tablet_bootstrap.cc:492] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab: Bootstrap starting.
I20260928 03:04:22.603102 9174 tablet_bootstrap.cc:654] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:22.604482 9224 ts_tablet_manager.cc:1918] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 03:04:22.604771 9224 log.cc:1196] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/0fc6a4d322c74202bc01ba02ef9606d3
I20260928 03:04:22.605448 9224 ts_tablet_manager.cc:1939] T 0fc6a4d322c74202bc01ba02ef9606d3 P 493349743b1a44ba839d4087a371c106: Deleting consensus metadata
I20260928 03:04:22.607671 8657 catalog_manager.cc:4841] TS 493349743b1a44ba839d4087a371c106 (127.6.192.1:37379): tablet 0fc6a4d322c74202bc01ba02ef9606d3 (table test-table [id=ffdd871f2e07411083d7bfac9d6b2401]) successfully deleted
I20260928 03:04:22.611538 9226 ts_tablet_manager.cc:1918] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 03:04:22.611753 9226 log.cc:1196] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/0fc6a4d322c74202bc01ba02ef9606d3
I20260928 03:04:22.612366 9226 ts_tablet_manager.cc:1939] T 0fc6a4d322c74202bc01ba02ef9606d3 P e5489b108dc84563bb1541d4a52e1a49: Deleting consensus metadata
I20260928 03:04:22.613368 9174 tablet_bootstrap.cc:492] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab: No bootstrap required, opened a new log
I20260928 03:04:22.613902 9174 ts_tablet_manager.cc:1397] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.007s
I20260928 03:04:22.614342 8654 catalog_manager.cc:4841] TS e5489b108dc84563bb1541d4a52e1a49 (127.6.192.3:46441): tablet 0fc6a4d322c74202bc01ba02ef9606d3 (table test-table [id=ffdd871f2e07411083d7bfac9d6b2401]) successfully deleted
I20260928 03:04:22.615361 9174 raft_consensus.cc:348] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.615795 9174 raft_consensus.cc:374] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:22.615940 9174 raft_consensus.cc:729] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6eb96db463547beb7a35b57f6bf6fab, State: Initialized, Role: FOLLOWER
I20260928 03:04:22.616289 9174 consensus_queue.cc:260] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.617182 9174 ts_tablet_manager.cc:1428] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 03:04:22.800093 9219 raft_consensus.cc:482] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:04:22.800793 9219 raft_consensus.cc:504] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.802953 9219 leader_election.cc:288] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 493349743b1a44ba839d4087a371c106 (127.6.192.1:37379), e5489b108dc84563bb1541d4a52e1a49 (127.6.192.3:46441)
I20260928 03:04:22.803735 9105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cd0631e5cc4e0c9751aa0730149bab" candidate_uuid: "d6eb96db463547beb7a35b57f6bf6fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5489b108dc84563bb1541d4a52e1a49" is_pre_election: true
I20260928 03:04:22.804044 9105 raft_consensus.cc:2454] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6eb96db463547beb7a35b57f6bf6fab in term 0.
I20260928 03:04:22.804656 8897 leader_election.cc:302] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6eb96db463547beb7a35b57f6bf6fab, e5489b108dc84563bb1541d4a52e1a49; no voters:
I20260928 03:04:22.804708 8826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cd0631e5cc4e0c9751aa0730149bab" candidate_uuid: "d6eb96db463547beb7a35b57f6bf6fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493349743b1a44ba839d4087a371c106" is_pre_election: true
I20260928 03:04:22.805013 8826 raft_consensus.cc:2454] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6eb96db463547beb7a35b57f6bf6fab in term 0.
I20260928 03:04:22.805373 9219 raft_consensus.cc:2781] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:04:22.805543 9219 raft_consensus.cc:482] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:04:22.805652 9219 raft_consensus.cc:3037] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:22.809098 9219 raft_consensus.cc:504] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.810680 9219 leader_election.cc:288] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [CANDIDATE]: Term 1 election: Requested vote from peers 493349743b1a44ba839d4087a371c106 (127.6.192.1:37379), e5489b108dc84563bb1541d4a52e1a49 (127.6.192.3:46441)
I20260928 03:04:22.810750 8826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cd0631e5cc4e0c9751aa0730149bab" candidate_uuid: "d6eb96db463547beb7a35b57f6bf6fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493349743b1a44ba839d4087a371c106"
I20260928 03:04:22.811036 8826 raft_consensus.cc:3037] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:22.814003 8826 raft_consensus.cc:2454] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6eb96db463547beb7a35b57f6bf6fab in term 1.
I20260928 03:04:22.814666 9105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cd0631e5cc4e0c9751aa0730149bab" candidate_uuid: "d6eb96db463547beb7a35b57f6bf6fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5489b108dc84563bb1541d4a52e1a49"
I20260928 03:04:22.814700 8900 leader_election.cc:302] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493349743b1a44ba839d4087a371c106, d6eb96db463547beb7a35b57f6bf6fab; no voters:
I20260928 03:04:22.814955 9105 raft_consensus.cc:3037] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:22.815140 9219 raft_consensus.cc:2781] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:22.815284 9219 raft_consensus.cc:686] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [term 1 LEADER]: Becoming Leader. State: Replica: d6eb96db463547beb7a35b57f6bf6fab, State: Running, Role: LEADER
I20260928 03:04:22.815665 9219 consensus_queue.cc:237] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:22.819803 9105 raft_consensus.cc:2454] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6eb96db463547beb7a35b57f6bf6fab in term 1.
I20260928 03:04:22.824045 8670 catalog_manager.cc:5495] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab reported cstate change: term changed from 0 to 1, leader changed from <none> to d6eb96db463547beb7a35b57f6bf6fab (127.6.192.2). New cstate: current_term: 1 leader_uuid: "d6eb96db463547beb7a35b57f6bf6fab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } health_report { overall_health: UNKNOWN } } }
I20260928 03:04:22.915654 9105 raft_consensus.cc:1261] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49 [term 1 FOLLOWER]: Refusing update from remote peer d6eb96db463547beb7a35b57f6bf6fab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:22.916612 9219 consensus_queue.cc:1035] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:04:22.916991 8826 raft_consensus.cc:1261] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106 [term 1 FOLLOWER]: Refusing update from remote peer d6eb96db463547beb7a35b57f6bf6fab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:22.920511 9219 consensus_queue.cc:1035] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [LEADER]: Connected to new peer: Peer: permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 03:04:22.945230 9205 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 03:04:24.555315 8670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:04:24.555727 8670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:04:24.560523 8670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf: Sending DeleteTablet for 3 replicas of tablet 66cd0631e5cc4e0c9751aa0730149bab
I20260928 03:04:24.561466 8807 tablet_service.cc:1514] Processing DeleteTablet for tablet 66cd0631e5cc4e0c9751aa0730149bab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:24 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:50162
I20260928 03:04:24.561510 8946 tablet_service.cc:1514] Processing DeleteTablet for tablet 66cd0631e5cc4e0c9751aa0730149bab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:24 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:55346
I20260928 03:04:24.561627 9085 tablet_service.cc:1514] Processing DeleteTablet for tablet 66cd0631e5cc4e0c9751aa0730149bab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:24 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:49088
I20260928 03:04:24.574568 9256 tablet_replica.cc:330] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab: stopping tablet replica
I20260928 03:04:24.575021 9257 tablet_replica.cc:330] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49: stopping tablet replica
I20260928 03:04:24.575052 9256 raft_consensus.cc:2229] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:04:24.575244 9255 tablet_replica.cc:330] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106: stopping tablet replica
I20260928 03:04:24.575273 9257 raft_consensus.cc:2229] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:04:24.575495 9255 raft_consensus.cc:2229] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:04:24.575563 9257 raft_consensus.cc:2258] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:24.575842 9256 raft_consensus.cc:2258] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:24.576216 9255 raft_consensus.cc:2258] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:24.576959 9257 ts_tablet_manager.cc:1905] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:24.576972 9256 ts_tablet_manager.cc:1905] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:24.577312 9255 ts_tablet_manager.cc:1905] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:24.585044 9256 ts_tablet_manager.cc:1918] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 03:04:24.585039 9255 ts_tablet_manager.cc:1918] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 03:04:24.585039 9257 ts_tablet_manager.cc:1918] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 03:04:24.585224 9256 log.cc:1196] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/66cd0631e5cc4e0c9751aa0730149bab
I20260928 03:04:24.585244 9255 log.cc:1196] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/66cd0631e5cc4e0c9751aa0730149bab
I20260928 03:04:24.585409 9257 log.cc:1196] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/66cd0631e5cc4e0c9751aa0730149bab
I20260928 03:04:24.585703 9256 ts_tablet_manager.cc:1939] T 66cd0631e5cc4e0c9751aa0730149bab P d6eb96db463547beb7a35b57f6bf6fab: Deleting consensus metadata
I20260928 03:04:24.585767 9255 ts_tablet_manager.cc:1939] T 66cd0631e5cc4e0c9751aa0730149bab P 493349743b1a44ba839d4087a371c106: Deleting consensus metadata
I20260928 03:04:24.586107 9257 ts_tablet_manager.cc:1939] T 66cd0631e5cc4e0c9751aa0730149bab P e5489b108dc84563bb1541d4a52e1a49: Deleting consensus metadata
I20260928 03:04:24.587139 8657 catalog_manager.cc:4841] TS 493349743b1a44ba839d4087a371c106 (127.6.192.1:37379): tablet 66cd0631e5cc4e0c9751aa0730149bab (table test-table [id=57743fba5c9d4a939650e08488d70b74]) successfully deleted
I20260928 03:04:24.587743 8654 catalog_manager.cc:4841] TS e5489b108dc84563bb1541d4a52e1a49 (127.6.192.3:46441): tablet 66cd0631e5cc4e0c9751aa0730149bab (table test-table [id=57743fba5c9d4a939650e08488d70b74]) successfully deleted
I20260928 03:04:24.587911 8657 catalog_manager.cc:4841] TS d6eb96db463547beb7a35b57f6bf6fab (127.6.192.2:46685): tablet 66cd0631e5cc4e0c9751aa0730149bab (table test-table [id=57743fba5c9d4a939650e08488d70b74]) successfully deleted
Sep 28 03:04:24 dist-test-slave-fjww krb5kdc[8625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564664, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 03:04:24 dist-test-slave-fjww krb5kdc[8625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564664, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:24.625977 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:24.610471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49562 (local address 127.6.192.62:38929)
0928 03:04:24.611064 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:24.611069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:24.611093 (+ 24us) server_negotiation.cc:407] Connection header received
0928 03:04:24.611155 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:24.611159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:24.611220 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:24.611317 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:24.612733 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:24.613584 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:24.614895 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:24.615310 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:24.615410 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:24.620299 (+ 4889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:24.620304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:24.620307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:24.620346 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:24.623111 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:24.623921 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:24.623925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:24.623927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:24.624065 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:24.624509 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:24.624512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:24.624514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:24.624808 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:24.624988 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:24.625335 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 03:04:24.625550 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":302,"threads_started":1}
I20260928 03:04:24.629840 8670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:04:24.630734 8670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:04:24.639565 8807 tablet_service.cc:1467] Processing CreateTablet for tablet da19f2deac634d789704832147b46af5 (DEFAULT_TABLE table=test-table [id=66da00318a564085842fef9e7e877625]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:24.639755 9085 tablet_service.cc:1467] Processing CreateTablet for tablet da19f2deac634d789704832147b46af5 (DEFAULT_TABLE table=test-table [id=66da00318a564085842fef9e7e877625]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:24.639894 8946 tablet_service.cc:1467] Processing CreateTablet for tablet da19f2deac634d789704832147b46af5 (DEFAULT_TABLE table=test-table [id=66da00318a564085842fef9e7e877625]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:24.640081 8807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da19f2deac634d789704832147b46af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:24.640141 9085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da19f2deac634d789704832147b46af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:24.640367 8946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da19f2deac634d789704832147b46af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:24.648140 9268 tablet_bootstrap.cc:492] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49: Bootstrap starting.
I20260928 03:04:24.648931 9267 tablet_bootstrap.cc:492] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106: Bootstrap starting.
I20260928 03:04:24.650610 9268 tablet_bootstrap.cc:654] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:24.651525 9267 tablet_bootstrap.cc:654] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:24.653024 9268 tablet_bootstrap.cc:492] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49: No bootstrap required, opened a new log
I20260928 03:04:24.653177 9268 ts_tablet_manager.cc:1397] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 03:04:24.653211 9266 tablet_bootstrap.cc:492] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab: Bootstrap starting.
I20260928 03:04:24.653975 9268 raft_consensus.cc:348] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:24.654158 9267 tablet_bootstrap.cc:492] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106: No bootstrap required, opened a new log
I20260928 03:04:24.654237 9268 raft_consensus.cc:374] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:24.654313 9267 ts_tablet_manager.cc:1397] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 03:04:24.654348 9268 raft_consensus.cc:729] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5489b108dc84563bb1541d4a52e1a49, State: Initialized, Role: FOLLOWER
I20260928 03:04:24.654608 9268 consensus_queue.cc:260] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:24.655179 9268 ts_tablet_manager.cc:1428] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260928 03:04:24.655230 9266 tablet_bootstrap.cc:654] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:24.655097 9267 raft_consensus.cc:348] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:24.655347 9267 raft_consensus.cc:374] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:24.655473 9267 raft_consensus.cc:729] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493349743b1a44ba839d4087a371c106, State: Initialized, Role: FOLLOWER
I20260928 03:04:24.655764 9267 consensus_queue.cc:260] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:24.656754 9267 ts_tablet_manager.cc:1428] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 03:04:24.660905 9266 tablet_bootstrap.cc:492] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab: No bootstrap required, opened a new log
I20260928 03:04:24.661084 9266 ts_tablet_manager.cc:1397] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260928 03:04:24.661816 9266 raft_consensus.cc:348] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:24.662038 9266 raft_consensus.cc:374] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:24.662156 9266 raft_consensus.cc:729] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6eb96db463547beb7a35b57f6bf6fab, State: Initialized, Role: FOLLOWER
I20260928 03:04:24.662397 9266 consensus_queue.cc:260] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:24.663373 9266 ts_tablet_manager.cc:1428] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 03:04:24.833217 9246 raft_consensus.cc:482] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:04:24.833568 9246 raft_consensus.cc:504] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:24.834342 9246 leader_election.cc:288] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 493349743b1a44ba839d4087a371c106 (127.6.192.1:37379), e5489b108dc84563bb1541d4a52e1a49 (127.6.192.3:46441)
I20260928 03:04:24.834985 8826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da19f2deac634d789704832147b46af5" candidate_uuid: "d6eb96db463547beb7a35b57f6bf6fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493349743b1a44ba839d4087a371c106" is_pre_election: true
I20260928 03:04:24.835036 9105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da19f2deac634d789704832147b46af5" candidate_uuid: "d6eb96db463547beb7a35b57f6bf6fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5489b108dc84563bb1541d4a52e1a49" is_pre_election: true
I20260928 03:04:24.835309 8826 raft_consensus.cc:2454] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6eb96db463547beb7a35b57f6bf6fab in term 0.
I20260928 03:04:24.835356 9105 raft_consensus.cc:2454] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6eb96db463547beb7a35b57f6bf6fab in term 0.
I20260928 03:04:24.835881 8900 leader_election.cc:302] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493349743b1a44ba839d4087a371c106, d6eb96db463547beb7a35b57f6bf6fab; no voters:
I20260928 03:04:24.836158 9246 raft_consensus.cc:2781] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:04:24.836297 9246 raft_consensus.cc:482] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:04:24.836416 9246 raft_consensus.cc:3037] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:24.838532 9246 raft_consensus.cc:504] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:24.839181 9246 leader_election.cc:288] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [CANDIDATE]: Term 1 election: Requested vote from peers 493349743b1a44ba839d4087a371c106 (127.6.192.1:37379), e5489b108dc84563bb1541d4a52e1a49 (127.6.192.3:46441)
I20260928 03:04:24.839829 9105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da19f2deac634d789704832147b46af5" candidate_uuid: "d6eb96db463547beb7a35b57f6bf6fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5489b108dc84563bb1541d4a52e1a49"
I20260928 03:04:24.839815 8826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da19f2deac634d789704832147b46af5" candidate_uuid: "d6eb96db463547beb7a35b57f6bf6fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493349743b1a44ba839d4087a371c106"
I20260928 03:04:24.840056 9105 raft_consensus.cc:3037] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:24.840098 8826 raft_consensus.cc:3037] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:24.842649 9105 raft_consensus.cc:2454] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6eb96db463547beb7a35b57f6bf6fab in term 1.
I20260928 03:04:24.842653 8826 raft_consensus.cc:2454] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6eb96db463547beb7a35b57f6bf6fab in term 1.
I20260928 03:04:24.843237 8900 leader_election.cc:302] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493349743b1a44ba839d4087a371c106, d6eb96db463547beb7a35b57f6bf6fab; no voters:
I20260928 03:04:24.843643 9246 raft_consensus.cc:2781] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:24.843798 9246 raft_consensus.cc:686] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [term 1 LEADER]: Becoming Leader. State: Replica: d6eb96db463547beb7a35b57f6bf6fab, State: Running, Role: LEADER
I20260928 03:04:24.844094 9246 consensus_queue.cc:237] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:24.847147 8670 catalog_manager.cc:5495] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab reported cstate change: term changed from 0 to 1, leader changed from <none> to d6eb96db463547beb7a35b57f6bf6fab (127.6.192.2). New cstate: current_term: 1 leader_uuid: "d6eb96db463547beb7a35b57f6bf6fab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } health_report { overall_health: UNKNOWN } } }
I20260928 03:04:24.909519 9276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:24.903647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55368 (local address 127.6.192.2:46685)
0928 03:04:24.904238 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:24.904243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:24.904264 (+ 21us) server_negotiation.cc:407] Connection header received
0928 03:04:24.904328 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:24.904331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:24.904396 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:24.904488 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:04:24.904908 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:24.905789 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:24.907319 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:24.907764 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:24.907878 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:24.908558 (+ 680us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:04:24.908690 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:04:24.908791 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:24.908955 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 03:04:24.909030 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":315,"threads_started":1}
I20260928 03:04:24.921036 8826 raft_consensus.cc:1261] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106 [term 1 FOLLOWER]: Refusing update from remote peer d6eb96db463547beb7a35b57f6bf6fab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:24.921110 9105 raft_consensus.cc:1261] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49 [term 1 FOLLOWER]: Refusing update from remote peer d6eb96db463547beb7a35b57f6bf6fab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:24.921677 9249 consensus_queue.cc:1035] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [LEADER]: Connected to new peer: Peer: permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:04:24.922078 9246 consensus_queue.cc:1035] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:04:24.970970 8670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:04:24.971261 8670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:04:24.974400 8670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf: Sending DeleteTablet for 3 replicas of tablet da19f2deac634d789704832147b46af5
I20260928 03:04:24.975375 9085 tablet_service.cc:1514] Processing DeleteTablet for tablet da19f2deac634d789704832147b46af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:24 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:49088
I20260928 03:04:24.975356 8807 tablet_service.cc:1514] Processing DeleteTablet for tablet da19f2deac634d789704832147b46af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:24 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:50162
I20260928 03:04:24.975375 8946 tablet_service.cc:1514] Processing DeleteTablet for tablet da19f2deac634d789704832147b46af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:24 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:55346
I20260928 03:04:24.975752 9257 tablet_replica.cc:330] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49: stopping tablet replica
I20260928 03:04:24.975755 9255 tablet_replica.cc:330] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106: stopping tablet replica
I20260928 03:04:24.975764 9256 tablet_replica.cc:330] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab: stopping tablet replica
I20260928 03:04:24.975953 9255 raft_consensus.cc:2229] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:04:24.975978 9256 raft_consensus.cc:2229] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:04:24.975996 9257 raft_consensus.cc:2229] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:04:24.976168 9257 raft_consensus.cc:2258] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:24.976158 9255 raft_consensus.cc:2258] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:24.976300 9256 raft_consensus.cc:2258] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:24.977187 8670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 03:04:24.977458 9255 ts_tablet_manager.cc:1905] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:24.977447 9256 ts_tablet_manager.cc:1905] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:24.977473 9257 ts_tablet_manager.cc:1905] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 03:04:24.977947 8670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:04:24.984478 9257 ts_tablet_manager.cc:1918] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 03:04:24.984530 9255 ts_tablet_manager.cc:1918] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 03:04:24.984700 9255 log.cc:1196] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/da19f2deac634d789704832147b46af5
I20260928 03:04:24.984838 9256 ts_tablet_manager.cc:1918] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 03:04:24.984882 9257 log.cc:1196] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/da19f2deac634d789704832147b46af5
I20260928 03:04:24.985005 9256 log.cc:1196] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/da19f2deac634d789704832147b46af5
I20260928 03:04:24.985167 9255 ts_tablet_manager.cc:1939] T da19f2deac634d789704832147b46af5 P 493349743b1a44ba839d4087a371c106: Deleting consensus metadata
I20260928 03:04:24.985462 9256 ts_tablet_manager.cc:1939] T da19f2deac634d789704832147b46af5 P d6eb96db463547beb7a35b57f6bf6fab: Deleting consensus metadata
I20260928 03:04:24.986086 8807 tablet_service.cc:1467] Processing CreateTablet for tablet 2e81f5e76c6e488bb960839a0a2c1322 (DEFAULT_TABLE table=test-table [id=3257bf138ac6499ca6ff3970683c5703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:24.986076 9085 tablet_service.cc:1467] Processing CreateTablet for tablet 2e81f5e76c6e488bb960839a0a2c1322 (DEFAULT_TABLE table=test-table [id=3257bf138ac6499ca6ff3970683c5703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:24.986438 8946 tablet_service.cc:1467] Processing CreateTablet for tablet 2e81f5e76c6e488bb960839a0a2c1322 (DEFAULT_TABLE table=test-table [id=3257bf138ac6499ca6ff3970683c5703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:24.986542 8807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e81f5e76c6e488bb960839a0a2c1322. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:24.986624 9085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e81f5e76c6e488bb960839a0a2c1322. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:24.986851 8946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e81f5e76c6e488bb960839a0a2c1322. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:24.987654 9257 ts_tablet_manager.cc:1939] T da19f2deac634d789704832147b46af5 P e5489b108dc84563bb1541d4a52e1a49: Deleting consensus metadata
I20260928 03:04:24.989228 8657 catalog_manager.cc:4841] TS d6eb96db463547beb7a35b57f6bf6fab (127.6.192.2:46685): tablet da19f2deac634d789704832147b46af5 (table test-table [id=66da00318a564085842fef9e7e877625]) successfully deleted
I20260928 03:04:24.989526 8657 catalog_manager.cc:4841] TS 493349743b1a44ba839d4087a371c106 (127.6.192.1:37379): tablet da19f2deac634d789704832147b46af5 (table test-table [id=66da00318a564085842fef9e7e877625]) successfully deleted
I20260928 03:04:24.989696 8654 catalog_manager.cc:4841] TS e5489b108dc84563bb1541d4a52e1a49 (127.6.192.3:46441): tablet da19f2deac634d789704832147b46af5 (table test-table [id=66da00318a564085842fef9e7e877625]) successfully deleted
I20260928 03:04:24.992749 9267 tablet_bootstrap.cc:492] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106: Bootstrap starting.
I20260928 03:04:24.993163 9266 tablet_bootstrap.cc:492] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab: Bootstrap starting.
I20260928 03:04:24.993923 9268 tablet_bootstrap.cc:492] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49: Bootstrap starting.
I20260928 03:04:24.994479 9267 tablet_bootstrap.cc:654] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:24.994709 9266 tablet_bootstrap.cc:654] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:24.995724 9268 tablet_bootstrap.cc:654] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:24.996953 9266 tablet_bootstrap.cc:492] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab: No bootstrap required, opened a new log
I20260928 03:04:24.997131 9266 ts_tablet_manager.cc:1397] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 03:04:24.997816 9266 raft_consensus.cc:348] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:24.997972 9268 tablet_bootstrap.cc:492] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49: No bootstrap required, opened a new log
I20260928 03:04:24.998010 9266 raft_consensus.cc:374] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:24.998063 9266 raft_consensus.cc:729] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6eb96db463547beb7a35b57f6bf6fab, State: Initialized, Role: FOLLOWER
I20260928 03:04:24.998117 9268 ts_tablet_manager.cc:1397] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 03:04:24.998306 9266 consensus_queue.cc:260] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:24.998852 9266 ts_tablet_manager.cc:1428] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 03:04:24.998901 9268 raft_consensus.cc:348] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:24.999136 9268 raft_consensus.cc:374] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:24.999220 9268 raft_consensus.cc:729] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5489b108dc84563bb1541d4a52e1a49, State: Initialized, Role: FOLLOWER
I20260928 03:04:24.999516 9268 consensus_queue.cc:260] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:25.000543 9267 tablet_bootstrap.cc:492] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106: No bootstrap required, opened a new log
I20260928 03:04:25.000741 9267 ts_tablet_manager.cc:1397] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20260928 03:04:25.000808 9268 ts_tablet_manager.cc:1428] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 03:04:25.001606 9267 raft_consensus.cc:348] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:25.001888 9267 raft_consensus.cc:374] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:25.002002 9267 raft_consensus.cc:729] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493349743b1a44ba839d4087a371c106, State: Initialized, Role: FOLLOWER
I20260928 03:04:25.002278 9267 consensus_queue.cc:260] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:25.002883 9267 ts_tablet_manager.cc:1428] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 03:04:25.181501 9274 raft_consensus.cc:482] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:04:25.181720 9274 raft_consensus.cc:504] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:25.182351 9274 leader_election.cc:288] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 493349743b1a44ba839d4087a371c106 (127.6.192.1:37379), d6eb96db463547beb7a35b57f6bf6fab (127.6.192.2:46685)
I20260928 03:04:25.182987 8827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e81f5e76c6e488bb960839a0a2c1322" candidate_uuid: "e5489b108dc84563bb1541d4a52e1a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493349743b1a44ba839d4087a371c106" is_pre_election: true
I20260928 03:04:25.183097 8971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e81f5e76c6e488bb960839a0a2c1322" candidate_uuid: "e5489b108dc84563bb1541d4a52e1a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6eb96db463547beb7a35b57f6bf6fab" is_pre_election: true
I20260928 03:04:25.183260 8827 raft_consensus.cc:2454] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5489b108dc84563bb1541d4a52e1a49 in term 0.
I20260928 03:04:25.183396 8971 raft_consensus.cc:2454] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5489b108dc84563bb1541d4a52e1a49 in term 0.
I20260928 03:04:25.183830 9039 leader_election.cc:302] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493349743b1a44ba839d4087a371c106, e5489b108dc84563bb1541d4a52e1a49; no voters:
I20260928 03:04:25.184101 9274 raft_consensus.cc:2781] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:04:25.184222 9274 raft_consensus.cc:482] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:04:25.184342 9274 raft_consensus.cc:3037] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:25.186455 9274 raft_consensus.cc:504] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:25.187096 9274 leader_election.cc:288] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [CANDIDATE]: Term 1 election: Requested vote from peers 493349743b1a44ba839d4087a371c106 (127.6.192.1:37379), d6eb96db463547beb7a35b57f6bf6fab (127.6.192.2:46685)
I20260928 03:04:25.187789 8827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e81f5e76c6e488bb960839a0a2c1322" candidate_uuid: "e5489b108dc84563bb1541d4a52e1a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493349743b1a44ba839d4087a371c106"
I20260928 03:04:25.188007 8827 raft_consensus.cc:3037] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:25.187919 8971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e81f5e76c6e488bb960839a0a2c1322" candidate_uuid: "e5489b108dc84563bb1541d4a52e1a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6eb96db463547beb7a35b57f6bf6fab"
I20260928 03:04:25.188148 8971 raft_consensus.cc:3037] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:25.189972 8827 raft_consensus.cc:2454] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5489b108dc84563bb1541d4a52e1a49 in term 1.
I20260928 03:04:25.190217 8971 raft_consensus.cc:2454] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5489b108dc84563bb1541d4a52e1a49 in term 1.
I20260928 03:04:25.190438 9039 leader_election.cc:302] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493349743b1a44ba839d4087a371c106, e5489b108dc84563bb1541d4a52e1a49; no voters:
I20260928 03:04:25.190794 9274 raft_consensus.cc:2781] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:25.191011 9274 raft_consensus.cc:686] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [term 1 LEADER]: Becoming Leader. State: Replica: e5489b108dc84563bb1541d4a52e1a49, State: Running, Role: LEADER
I20260928 03:04:25.191305 9274 consensus_queue.cc:237] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } }
I20260928 03:04:25.194111 8670 catalog_manager.cc:5495] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5489b108dc84563bb1541d4a52e1a49 (127.6.192.3). New cstate: current_term: 1 leader_uuid: "e5489b108dc84563bb1541d4a52e1a49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5489b108dc84563bb1541d4a52e1a49" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46441 } health_report { overall_health: HEALTHY } } }
I20260928 03:04:25.216557 9284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:25.211056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49110 (local address 127.6.192.3:46441)
0928 03:04:25.211884 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:25.211895 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:25.211913 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:04:25.211971 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:25.211974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:25.212035 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:25.212155 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:04:25.212568 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:25.213322 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:25.214648 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:25.215081 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:25.215228 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:25.215797 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:04:25.215912 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:04:25.216022 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:25.216144 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 03:04:25.216221 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":561,"threads_started":1}
I20260928 03:04:25.232717 9284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:25.226371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:55601 (local address 127.6.192.3:46441)
0928 03:04:25.226589 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:25.226596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:25.227193 (+ 597us) server_negotiation.cc:407] Connection header received
0928 03:04:25.227350 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:25.227357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:25.227434 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:25.227547 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:04:25.227988 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:25.229054 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:25.230511 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:25.231157 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:25.231280 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:25.231831 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:04:25.231961 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:04:25.232042 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:25.232239 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 03:04:25.232324 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 03:04:25.232753 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:25.226354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:46441 (local address 127.6.192.3:55601)
0928 03:04:25.227095 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:25.227111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:25.227231 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:25.227560 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:25.227563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:25.227573 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 03:04:25.227812 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:25.227820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:25.229293 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:25.229300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:25.230369 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:25.230381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:25.231322 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:25.231326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:25.231578 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:25.231645 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 03:04:25.232062 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 03:04:25.232071 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 03:04:25.232207 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 03:04:25.232347 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":404,"threads_started":1}
I20260928 03:04:25.242017 8971 raft_consensus.cc:1261] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab [term 1 FOLLOWER]: Refusing update from remote peer e5489b108dc84563bb1541d4a52e1a49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:25.242031 8827 raft_consensus.cc:1261] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106 [term 1 FOLLOWER]: Refusing update from remote peer e5489b108dc84563bb1541d4a52e1a49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:25.242620 9236 consensus_queue.cc:1035] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6eb96db463547beb7a35b57f6bf6fab" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:04:25.243088 9233 consensus_queue.cc:1035] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "493349743b1a44ba839d4087a371c106" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 03:04:25.267498 9234 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 03:04:26.860136 8670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:04:26.860425 8670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:04:26.864923 8670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6df20f9377841bba26fc46273e484cf: Sending DeleteTablet for 3 replicas of tablet 2e81f5e76c6e488bb960839a0a2c1322
I20260928 03:04:26.865993 8946 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e81f5e76c6e488bb960839a0a2c1322 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:26 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:55346
I20260928 03:04:26.866008 8807 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e81f5e76c6e488bb960839a0a2c1322 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:26 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:50162
I20260928 03:04:26.866122 9084 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e81f5e76c6e488bb960839a0a2c1322 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:26 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:49088
I20260928 03:04:26.867048 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 8741
I20260928 03:04:26.883723 9313 tablet_replica.cc:330] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106: stopping tablet replica
I20260928 03:04:26.884083 9313 raft_consensus.cc:2229] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:04:26.884459 9313 raft_consensus.cc:2258] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:26.885207 9312 tablet_replica.cc:330] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49: stopping tablet replica
I20260928 03:04:26.885469 9312 raft_consensus.cc:2229] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:04:26.885550 9311 tablet_replica.cc:330] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab: stopping tablet replica
I20260928 03:04:26.885767 9311 raft_consensus.cc:2229] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:04:26.886046 9311 raft_consensus.cc:2258] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:26.886178 9312 raft_consensus.cc:2258] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:26.887151 9313 ts_tablet_manager.cc:1905] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:26.887256 9311 ts_tablet_manager.cc:1905] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:26.887662 9312 ts_tablet_manager.cc:1905] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 03:04:26 dist-test-slave-fjww krb5kdc[8625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564664, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:04:26.894960 9311 ts_tablet_manager.cc:1918] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 03:04:26.895212 9311 log.cc:1196] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/2e81f5e76c6e488bb960839a0a2c1322
I20260928 03:04:26.895394 9312 ts_tablet_manager.cc:1918] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 03:04:26.895582 9312 log.cc:1196] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/2e81f5e76c6e488bb960839a0a2c1322
I20260928 03:04:26.895623 9313 ts_tablet_manager.cc:1918] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 03:04:26.895789 9313 log.cc:1196] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/2e81f5e76c6e488bb960839a0a2c1322
I20260928 03:04:26.896185 9311 ts_tablet_manager.cc:1939] T 2e81f5e76c6e488bb960839a0a2c1322 P d6eb96db463547beb7a35b57f6bf6fab: Deleting consensus metadata
I20260928 03:04:26.896193 9312 ts_tablet_manager.cc:1939] T 2e81f5e76c6e488bb960839a0a2c1322 P e5489b108dc84563bb1541d4a52e1a49: Deleting consensus metadata
I20260928 03:04:26.896399 9313 ts_tablet_manager.cc:1939] T 2e81f5e76c6e488bb960839a0a2c1322 P 493349743b1a44ba839d4087a371c106: Deleting consensus metadata
I20260928 03:04:26.897575 8654 catalog_manager.cc:4841] TS e5489b108dc84563bb1541d4a52e1a49 (127.6.192.3:46441): tablet 2e81f5e76c6e488bb960839a0a2c1322 (table test-table [id=3257bf138ac6499ca6ff3970683c5703]) successfully deleted
I20260928 03:04:26.897601 8657 catalog_manager.cc:4841] TS d6eb96db463547beb7a35b57f6bf6fab (127.6.192.2:46685): tablet 2e81f5e76c6e488bb960839a0a2c1322 (table test-table [id=3257bf138ac6499ca6ff3970683c5703]) successfully deleted
I20260928 03:04:26.897966 8657 catalog_manager.cc:4841] TS 493349743b1a44ba839d4087a371c106 (127.6.192.1:37379): tablet 2e81f5e76c6e488bb960839a0a2c1322 (table test-table [id=3257bf138ac6499ca6ff3970683c5703]) successfully deleted
I20260928 03:04:26.902864 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:26.869145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50166 (local address 127.6.192.1:37379)
0928 03:04:26.876047 (+ 6902us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:26.876059 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:26.881983 (+ 5924us) server_negotiation.cc:407] Connection header received
0928 03:04:26.882616 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:26.882626 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:26.882744 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:26.882848 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:26.886146 (+ 3298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:26.887320 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:26.889316 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:26.890034 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:26.890139 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:26.896255 (+ 6116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:26.896282 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:26.896306 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:26.896367 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:26.899988 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:26.900792 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:26.900799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:26.900802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:26.900970 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:26.901355 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:26.901360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:26.901363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:26.901716 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:26.901971 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:26.902288 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 03:04:26.902468 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6672,"thread_start_us":6443,"threads_started":1}
W20260928 03:04:26.903728 8872 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50166
W20260928 03:04:26.904523 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 8741: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:04:26.904724 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 8741
I20260928 03:04:26.923064 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 8881
Sep 28 03:04:26 dist-test-slave-fjww krb5kdc[8625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564664, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.192.2@KRBTEST.COM
I20260928 03:04:26.940475 9316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:26.923981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55378 (local address 127.6.192.2:46685)
0928 03:04:26.924578 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:26.924582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:26.924598 (+ 16us) server_negotiation.cc:407] Connection header received
0928 03:04:26.924651 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:26.924655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:26.924723 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:26.924849 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:26.926152 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:26.926939 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:26.928303 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:26.928760 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:26.928914 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:26.933976 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:26.933994 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:26.934010 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:26.934051 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:26.937656 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:26.938453 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:26.938461 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:26.938463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:26.938608 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:26.939000 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:26.939004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:26.939006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:26.939379 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:26.939723 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:26.940020 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 03:04:26.940206 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":266,"threads_started":1}
W20260928 03:04:26.941250 9011 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55378
W20260928 03:04:26.941972 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 8881: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:04:26.942205 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 8881
I20260928 03:04:26.964826 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 9020
Sep 28 03:04:26 dist-test-slave-fjww krb5kdc[8625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564664, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.192.3@KRBTEST.COM
I20260928 03:04:26.981485 9317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:26.965692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49122 (local address 127.6.192.3:46441)
0928 03:04:26.966236 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:26.966241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:26.966258 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:04:26.966312 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:26.966316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:26.966375 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:26.966475 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:26.967923 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:26.969044 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:26.970453 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:26.970893 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:26.971025 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:26.975550 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:26.975568 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:26.975584 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:26.975633 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:26.978643 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:26.979386 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:26.979395 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:26.979397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:26.979602 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:26.979983 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:26.979987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:26.979989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:26.980336 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:26.980572 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:26.980968 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 03:04:26.981159 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":308,"threads_started":1}
W20260928 03:04:26.982169 9150 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49122
W20260928 03:04:26.982795 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 9020: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:04:26.982980 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 9020
I20260928 03:04:27.008826 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 8638
I20260928 03:04:27.010216 8710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:27.131798 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 8638
2026-09-28T03:04:27Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8930 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:04:27 dist-test-slave-fjww krb5kdc[9321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:04:27 dist-test-slave-fjww krb5kdc[9321](info): set up 2 sockets
Sep 28 03:04:27 dist-test-slave-fjww krb5kdc[9321](info): commencing operation
krb5kdc: starting...
W20260928 03:04:28.256083 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:04:28 dist-test-slave-fjww krb5kdc[9321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:04:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:04:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:04:28.477649 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:38075
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:38075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:28.824229 9334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:28.824589 9334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:28.824759 9334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:28.832373 9334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:04:28.832497 9334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:28.832587 9334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:28.832701 9334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:04:28.832767 9334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:04:28.843359 9334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:38075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:38075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:28.844803 9334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:28.846222 9334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:28.859949 9340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:28.859879 9342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:28.860004 9339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:28.860261 9334 server_base.cc:1034] running on GCE node
I20260928 03:04:28.861596 9334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:28.863200 9334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:28.864478 9334 hybrid_clock.cc:648] HybridClock initialized: now 1790564668864409 us; error 64 us; skew 500 ppm
I20260928 03:04:28.864886 9334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:28 dist-test-slave-fjww krb5kdc[9321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564668, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:28.872764 9334 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:04:28.874678 9334 webserver.cc:460] Webserver started at http://127.6.192.62:38271/ using document root <none> and password file <none>
I20260928 03:04:28.875578 9334 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:28.875746 9334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:28.876106 9334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:28.879423 9334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "0af9c196c641473bb64e057f0d723683"
format_stamp: "Formatted at 2026-09-28 03:04:28 on dist-test-slave-fjww"
server_key: "a57bbe87ed91846e112bbcca518ec49b"
server_key_iv: "04b2aa8a8d120ed07fee475ca819b7da"
server_key_version: "encryptionkey@0"
I20260928 03:04:28.880471 9334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "0af9c196c641473bb64e057f0d723683"
format_stamp: "Formatted at 2026-09-28 03:04:28 on dist-test-slave-fjww"
server_key: "a57bbe87ed91846e112bbcca518ec49b"
server_key_iv: "04b2aa8a8d120ed07fee475ca819b7da"
server_key_version: "encryptionkey@0"
I20260928 03:04:28.886094 9334 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 03:04:28.890033 9349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:28.891819 9334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:04:28.892045 9334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "0af9c196c641473bb64e057f0d723683"
format_stamp: "Formatted at 2026-09-28 03:04:28 on dist-test-slave-fjww"
server_key: "a57bbe87ed91846e112bbcca518ec49b"
server_key_iv: "04b2aa8a8d120ed07fee475ca819b7da"
server_key_version: "encryptionkey@0"
I20260928 03:04:28.892282 9334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:28.908406 9334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:28.913251 9334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:28.913589 9334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:28.933755 9334 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:38075
I20260928 03:04:28.933785 9400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:38075 every 8 connection(s)
I20260928 03:04:28.935321 9334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:04:28.939277 9401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:28.939885 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 9334
I20260928 03:04:28.940124 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:04:28.940927 6912 external_mini_cluster.cc:1428] Setting key 8f5194adc7bbae443b0196e07ba4eeb1
I20260928 03:04:28.953430 9401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683: Bootstrap starting.
Sep 28 03:04:28 dist-test-slave-fjww krb5kdc[9321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:28.957226 9401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:28.958467 9401 log.cc:824] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:28.962350 9401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683: No bootstrap required, opened a new log
I20260928 03:04:28.965679 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:28.943206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34994 (local address 127.6.192.62:38075)
0928 03:04:28.944059 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:28.944070 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:28.944111 (+ 41us) server_negotiation.cc:407] Connection header received
0928 03:04:28.946217 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:28.946242 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:28.946676 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:28.947092 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:28.948628 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:28.950016 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:28.951405 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:28.952175 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:28.952374 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:28.956858 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:28.956867 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:28.956885 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:28.956927 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:28.960913 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:28.961741 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:28.961751 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:28.961760 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:28.962015 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:28.962608 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:28.962614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:28.962617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:28.963235 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:28.963585 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:28.964094 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 03:04:28.964564 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":356,"threads_started":1}
I20260928 03:04:28.972059 9401 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af9c196c641473bb64e057f0d723683" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38075 } }
I20260928 03:04:28.972553 9401 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:28.972697 9401 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af9c196c641473bb64e057f0d723683, State: Initialized, Role: FOLLOWER
I20260928 03:04:28.973390 9401 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af9c196c641473bb64e057f0d723683" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38075 } }
I20260928 03:04:28.973690 9401 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:04:28.973860 9401 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:04:28.974061 9401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:28.976899 9401 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af9c196c641473bb64e057f0d723683" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38075 } }
I20260928 03:04:28.977463 9401 leader_election.cc:302] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0af9c196c641473bb64e057f0d723683; no voters:
I20260928 03:04:28.978138 9401 leader_election.cc:288] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:04:28.978242 9406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:28.979703 9406 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683 [term 1 LEADER]: Becoming Leader. State: Replica: 0af9c196c641473bb64e057f0d723683, State: Running, Role: LEADER
I20260928 03:04:28.980352 9406 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af9c196c641473bb64e057f0d723683" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38075 } }
I20260928 03:04:28.980729 9401 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:04:28.983426 9407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0af9c196c641473bb64e057f0d723683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af9c196c641473bb64e057f0d723683" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38075 } } }
I20260928 03:04:28.983671 9408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0af9c196c641473bb64e057f0d723683. Latest consensus state: current_term: 1 leader_uuid: "0af9c196c641473bb64e057f0d723683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af9c196c641473bb64e057f0d723683" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38075 } } }
I20260928 03:04:28.984092 9408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:28.983963 9407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:28.986006 9413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:04:28.993381 9413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:04:29.003491 9413 catalog_manager.cc:1347] Generated new cluster ID: 3c54a67c15284962b8c68e736df36363
I20260928 03:04:29.003619 9413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:04:29.021612 9413 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:04:29.023752 9413 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:04:29.038497 9413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0af9c196c641473bb64e057f0d723683: Generated new TSK 0
I20260928 03:04:29.039753 9413 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
I20260928 03:04:29.095773 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:38075
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:29.444625 9427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:29.444934 9427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:29.445030 9427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:29.452525 9427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:29.452649 9427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:29.452764 9427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:04:29.463148 9427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:38075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:29.464511 9427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:29.465911 9427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:29.480545 9432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:29.480567 9435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:29.480783 9433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:29.481506 9427 server_base.cc:1034] running on GCE node
I20260928 03:04:29.482208 9427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:29.483285 9427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:29.484570 9427 hybrid_clock.cc:648] HybridClock initialized: now 1790564669484494 us; error 65 us; skew 500 ppm
I20260928 03:04:29.484982 9427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:29 dist-test-slave-fjww krb5kdc[9321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564669, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:29.488896 9427 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:04:29.490514 9427 webserver.cc:460] Webserver started at http://127.6.192.1:35027/ using document root <none> and password file <none>
I20260928 03:04:29.491318 9427 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:29.491492 9427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:29.491882 9427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:29.494987 9427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "73b363f3869e41859d32c8a44e0b2d06"
format_stamp: "Formatted at 2026-09-28 03:04:29 on dist-test-slave-fjww"
server_key: "f9efbc26190dc60345e59d8a18238ee4"
server_key_iv: "dd7d7647da8aff839e04ccaba6feab25"
server_key_version: "encryptionkey@0"
I20260928 03:04:29.495945 9427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "73b363f3869e41859d32c8a44e0b2d06"
format_stamp: "Formatted at 2026-09-28 03:04:29 on dist-test-slave-fjww"
server_key: "f9efbc26190dc60345e59d8a18238ee4"
server_key_iv: "dd7d7647da8aff839e04ccaba6feab25"
server_key_version: "encryptionkey@0"
I20260928 03:04:29.501784 9427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:04:29.505755 9442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:29.507452 9427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 03:04:29.507679 9427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "73b363f3869e41859d32c8a44e0b2d06"
format_stamp: "Formatted at 2026-09-28 03:04:29 on dist-test-slave-fjww"
server_key: "f9efbc26190dc60345e59d8a18238ee4"
server_key_iv: "dd7d7647da8aff839e04ccaba6feab25"
server_key_version: "encryptionkey@0"
I20260928 03:04:29.507918 9427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:29.532344 9427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:29.537138 9427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:29.537477 9427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:29.538654 9427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:29.540132 9427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:29.540243 9427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:29.540408 9427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:29.540488 9427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:29.577171 9427 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:33777
I20260928 03:04:29.577359 9554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:33777 every 8 connection(s)
I20260928 03:04:29.578596 9427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:04:29.579155 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 9427
I20260928 03:04:29.579461 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:04:29.580401 6912 external_mini_cluster.cc:1428] Setting key d3c5960c3327ec296fcfb7a03209a4ce
Sep 28 03:04:29 dist-test-slave-fjww krb5kdc[9321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564669, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.192.2@KRBTEST.COM; defaulting to no policy
I20260928 03:04:29.602012 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:29.581679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:40937 (local address 127.6.192.62:38075)
0928 03:04:29.582177 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:29.582183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:29.583427 (+ 1244us) server_negotiation.cc:407] Connection header received
0928 03:04:29.585401 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:29.585406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:29.585497 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:29.585641 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:29.587261 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:29.588108 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:29.589680 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:29.590117 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:29.590268 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:29.594216 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:29.594226 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:29.594235 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:29.594295 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:29.598382 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:29.599295 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:29.599303 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:29.599309 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:29.599536 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:29.600046 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:29.600050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:29.600052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:29.600562 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:29.600674 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:29.601517 (+ 843us) server_negotiation.cc:299] Negotiation successful
0928 03:04:29.601742 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":254,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.2@KRBTEST.COM" created.
I20260928 03:04:29.603148 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:29.582366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:38075 (local address 127.6.192.1:40937)
0928 03:04:29.583253 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:29.583299 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:29.585091 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:29.585895 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:29.585904 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:29.586510 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:29.587083 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:29.587099 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:29.588284 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:29.588294 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:29.589527 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:29.589539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:29.590368 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:29.590378 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:29.590615 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:29.591357 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:29.591380 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:29.593935 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:29.598577 (+ 4642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:29.598585 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:29.598601 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:29.599159 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:29.599688 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:29.599692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:29.599694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:29.599921 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:29.600692 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:29.600699 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:29.601084 (+ 385us) client_negotiation.cc:769] Sending connection context
0928 03:04:29.601323 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 03:04:29.601739 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":402,"threads_started":1}
I20260928 03:04:29.604974 9555 heartbeater.cc:347] Connected to a master server at 127.6.192.62:38075
I20260928 03:04:29.605329 9555 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:29.606437 9555 heartbeater.cc:511] Master 127.6.192.62:38075 requested a full tablet report, sending...
I20260928 03:04:29.609198 9366 ts_manager.cc:194] Registered new tserver with Master: 73b363f3869e41859d32c8a44e0b2d06 (127.6.192.1:33777)
I20260928 03:04:29.611588 9366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:40937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
I20260928 03:04:29.631901 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:38075
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:29.974118 9561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:29.974426 9561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:29.974591 9561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:29.982935 9561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:29.983057 9561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:29.983273 9561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:04:29.993561 9561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:38075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:29.994946 9561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:29.996515 9561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:30.011373 9567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:30.011418 9566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:30.011934 9569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:30.012427 9561 server_base.cc:1034] running on GCE node
I20260928 03:04:30.013103 9561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:30.014166 9561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:30.015589 9561 hybrid_clock.cc:648] HybridClock initialized: now 1790564670015453 us; error 108 us; skew 500 ppm
I20260928 03:04:30.015996 9561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:30 dist-test-slave-fjww krb5kdc[9321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564670, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:30.019857 9561 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:04:30.021508 9561 webserver.cc:460] Webserver started at http://127.6.192.2:41691/ using document root <none> and password file <none>
I20260928 03:04:30.022244 9561 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:30.022404 9561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:30.022727 9561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:30.025897 9561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "90fa396bb41b4335b3dd4c93eb453278"
format_stamp: "Formatted at 2026-09-28 03:04:30 on dist-test-slave-fjww"
server_key: "22c01cb9ebe9e288cf399893504046db"
server_key_iv: "b3fd2585fc4ac277c17e7f965177f7c4"
server_key_version: "encryptionkey@0"
I20260928 03:04:30.026736 9561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "90fa396bb41b4335b3dd4c93eb453278"
format_stamp: "Formatted at 2026-09-28 03:04:30 on dist-test-slave-fjww"
server_key: "22c01cb9ebe9e288cf399893504046db"
server_key_iv: "b3fd2585fc4ac277c17e7f965177f7c4"
server_key_version: "encryptionkey@0"
I20260928 03:04:30.032351 9561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 03:04:30.036247 9576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:30.037712 9561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:04:30.037904 9561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "90fa396bb41b4335b3dd4c93eb453278"
format_stamp: "Formatted at 2026-09-28 03:04:30 on dist-test-slave-fjww"
server_key: "22c01cb9ebe9e288cf399893504046db"
server_key_iv: "b3fd2585fc4ac277c17e7f965177f7c4"
server_key_version: "encryptionkey@0"
I20260928 03:04:30.038128 9561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:30.055115 9561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:30.060055 9561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:30.060381 9561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:30.061481 9561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:30.062955 9561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:30.063064 9561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:30.063215 9561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:30.063289 9561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:30.099726 9561 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:45751
I20260928 03:04:30.099764 9688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:45751 every 8 connection(s)
I20260928 03:04:30.101157 9561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:04:30.102121 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 9561
I20260928 03:04:30.102334 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:04:30.102789 6912 external_mini_cluster.cc:1428] Setting key 08ea3693c1c3c8a2e513b2b97a6a6cf1
Sep 28 03:04:30 dist-test-slave-fjww krb5kdc[9321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564670, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.192.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.3@KRBTEST.COM" created.
I20260928 03:04:30.127317 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:30.104716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:48751 (local address 127.6.192.62:38075)
0928 03:04:30.105982 (+ 1266us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:30.105992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:30.106867 (+ 875us) server_negotiation.cc:407] Connection header received
0928 03:04:30.109745 (+ 2878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:30.109760 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:30.109865 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:30.110010 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:30.112026 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:30.113038 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:30.114833 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:30.115316 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:30.115527 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:30.119645 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:30.119651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:30.119658 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:30.119698 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:30.122923 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:30.124011 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:30.124017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:30.124020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:30.124213 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:30.124775 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:30.124783 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:30.124786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:30.125154 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:30.125364 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:30.126500 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0928 03:04:30.126815 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1122,"thread_start_us":943,"threads_started":1}
I20260928 03:04:30.128731 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:30.105474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:38075 (local address 127.6.192.2:48751)
0928 03:04:30.106684 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:30.106741 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:30.109325 (+ 2584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:30.110329 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:30.110339 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:30.110976 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:30.111818 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:30.111839 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:30.113245 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:30.113255 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:30.114637 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:30.114651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:30.115632 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:30.115640 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:30.115925 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:30.116717 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:30.116744 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:30.119311 (+ 2567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:30.123176 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:30.123185 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:30.123202 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:30.123838 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:30.124395 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:30.124398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:30.124401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:30.124621 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:30.125327 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:30.125336 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:30.125911 (+ 575us) client_negotiation.cc:769] Sending connection context
0928 03:04:30.126265 (+ 354us) client_negotiation.cc:240] Negotiation successful
0928 03:04:30.126979 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"thread_start_us":5599,"threads_started":1}
I20260928 03:04:30.130502 9689 heartbeater.cc:347] Connected to a master server at 127.6.192.62:38075
I20260928 03:04:30.130841 9689 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:30.131906 9689 heartbeater.cc:511] Master 127.6.192.62:38075 requested a full tablet report, sending...
I20260928 03:04:30.133817 9366 ts_manager.cc:194] Registered new tserver with Master: 90fa396bb41b4335b3dd4c93eb453278 (127.6.192.2:45751)
I20260928 03:04:30.134891 9366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:48751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
I20260928 03:04:30.156003 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:38075
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:30.502415 9695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:30.502804 9695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:30.502980 9695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:30.510656 9695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:30.510856 9695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:30.511092 9695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:04:30.521829 9695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:38075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:30.523267 9695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:30.524838 9695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:30.539906 9700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:30.539927 9703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:30.539989 9701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:30.540800 9695 server_base.cc:1034] running on GCE node
I20260928 03:04:30.541518 9695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:30.542716 9695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:30.543978 9695 hybrid_clock.cc:648] HybridClock initialized: now 1790564670543909 us; error 55 us; skew 500 ppm
I20260928 03:04:30.544420 9695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:30 dist-test-slave-fjww krb5kdc[9321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564670, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:30.548143 9695 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:04:30.549741 9695 webserver.cc:460] Webserver started at http://127.6.192.3:44311/ using document root <none> and password file <none>
I20260928 03:04:30.550472 9695 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:30.550637 9695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:30.551004 9695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:30.554127 9695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "7904d812ec2a4946b7421c540609fca3"
format_stamp: "Formatted at 2026-09-28 03:04:30 on dist-test-slave-fjww"
server_key: "ecd4b1cf57f1ec0a50d23e0f2ebc49dc"
server_key_iv: "d511c7b83ce9da662e1b9c0da395f0f7"
server_key_version: "encryptionkey@0"
I20260928 03:04:30.554987 9695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "7904d812ec2a4946b7421c540609fca3"
format_stamp: "Formatted at 2026-09-28 03:04:30 on dist-test-slave-fjww"
server_key: "ecd4b1cf57f1ec0a50d23e0f2ebc49dc"
server_key_iv: "d511c7b83ce9da662e1b9c0da395f0f7"
server_key_version: "encryptionkey@0"
I20260928 03:04:30.560761 9695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 03:04:30.564718 9710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:30.566308 9695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 03:04:30.566510 9695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "7904d812ec2a4946b7421c540609fca3"
format_stamp: "Formatted at 2026-09-28 03:04:30 on dist-test-slave-fjww"
server_key: "ecd4b1cf57f1ec0a50d23e0f2ebc49dc"
server_key_iv: "d511c7b83ce9da662e1b9c0da395f0f7"
server_key_version: "encryptionkey@0"
I20260928 03:04:30.566741 9695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:30.605432 9695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:30.610330 9695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:30.610649 9695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:30.612061 9695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:30.613602 9695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:30.613731 9695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:30.613881 9695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:30.613955 9695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:30.616036 9555 heartbeater.cc:503] Master 127.6.192.62:38075 was elected leader, sending a full tablet report...
I20260928 03:04:30.651580 9695 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:36123
I20260928 03:04:30.651676 9822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:36123 every 8 connection(s)
I20260928 03:04:30.653049 9695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:04:30.657110 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 9695
I20260928 03:04:30.657367 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:04:30.657979 6912 external_mini_cluster.cc:1428] Setting key c6fe9be57ddbc6207af81425049663f6
Sep 28 03:04:30 dist-test-slave-fjww krb5kdc[9321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564670, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:30.676430 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:30.656284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:59945 (local address 127.6.192.62:38075)
0928 03:04:30.656893 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:30.656903 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:30.657962 (+ 1059us) server_negotiation.cc:407] Connection header received
0928 03:04:30.660132 (+ 2170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:30.660137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:30.660229 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:30.660450 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:30.662560 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:30.663367 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:30.665040 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:30.665453 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:30.665555 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:30.669684 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:30.669690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:30.669693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:30.669735 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:30.672796 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:30.673754 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:30.673762 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:30.673769 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:30.673982 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:30.674546 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:30.674550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:30.674552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:30.674973 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:30.675142 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:30.675944 (+ 802us) server_negotiation.cc:299] Negotiation successful
0928 03:04:30.676124 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":1156,"threads_started":1}
I20260928 03:04:30.677570 9826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:30.656671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:38075 (local address 127.6.192.3:59945)
0928 03:04:30.657741 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:30.657799 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:30.659824 (+ 2025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:30.660787 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:30.660800 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:30.661717 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:30.662365 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:30.662389 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:30.663573 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:30.663577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:30.664867 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:30.664880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:30.665722 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:30.665728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:30.665946 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:30.666819 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:30.666844 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:30.669391 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:30.672948 (+ 3557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:30.672959 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:30.672980 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:30.673592 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:30.674120 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:30.674124 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:30.674127 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:30.674414 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:30.675120 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:30.675127 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:30.675580 (+ 453us) client_negotiation.cc:769] Sending connection context
0928 03:04:30.675852 (+ 272us) client_negotiation.cc:240] Negotiation successful
0928 03:04:30.676372 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":435,"threads_started":1}
I20260928 03:04:30.679167 9823 heartbeater.cc:347] Connected to a master server at 127.6.192.62:38075
I20260928 03:04:30.679612 9823 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:30.680439 9823 heartbeater.cc:511] Master 127.6.192.62:38075 requested a full tablet report, sending...
I20260928 03:04:30.682262 9366 ts_manager.cc:194] Registered new tserver with Master: 7904d812ec2a4946b7421c540609fca3 (127.6.192.3:36123)
I20260928 03:04:30.683189 9366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:59945
I20260928 03:04:30.687680 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 03:04:30.722146 9825 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:04:30.719474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35004 (local address 127.6.192.62:38075)
0928 03:04:30.719735 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:30.719746 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:30.720084 (+ 338us) server_negotiation.cc:407] Connection header received
0928 03:04:30.720311 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:30.720316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:30.720424 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:30.720551 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:30.721881 (+ 1330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35004: BlockingRecv error: recv got EOF from 127.0.0.1:35004 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 03:04:30.725631 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 9427
W20260928 03:04:30.728928 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 9427: Not authorized: Client connection negotiation failed: client connection to 127.6.192.1:33777: server requires authentication, but client does not have Kerberos credentials available
I20260928 03:04:30.729141 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 9427
I20260928 03:04:30.746045 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 9561
W20260928 03:04:30.749214 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 9561: Not authorized: Client connection negotiation failed: client connection to 127.6.192.2:45751: server requires authentication, but client does not have Kerberos credentials available
I20260928 03:04:30.749552 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 9561
I20260928 03:04:30.765198 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 9695
W20260928 03:04:30.768599 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 9695: Not authorized: Client connection negotiation failed: client connection to 127.6.192.3:36123: server requires authentication, but client does not have Kerberos credentials available
I20260928 03:04:30.768764 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 9695
I20260928 03:04:30.784799 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 9334
I20260928 03:04:30.786351 9396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:30.896220 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 9334
2026-09-28T03:04:30Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3761 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:04:30 dist-test-slave-fjww krb5kdc[9841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:04:30 dist-test-slave-fjww krb5kdc[9841](info): set up 2 sockets
Sep 28 03:04:30 dist-test-slave-fjww krb5kdc[9841](info): commencing operation
krb5kdc: starting...
W20260928 03:04:32.024477 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:04:32 dist-test-slave-fjww krb5kdc[9841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:04:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:04:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:04:32.247423 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:46721
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:46721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:32.583526 9854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:32.583910 9854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:32.584152 9854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:32.593098 9854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:04:32.593243 9854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:32.593297 9854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:32.593432 9854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:04:32.593600 9854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:04:32.606058 9854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:46721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:46721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:32.607676 9854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:32.609210 9854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:32.624228 9859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:32.624274 9860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:32.624297 9862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:32.625245 9854 server_base.cc:1034] running on GCE node
I20260928 03:04:32.626266 9854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:32.628127 9854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:32.629424 9854 hybrid_clock.cc:648] HybridClock initialized: now 1790564672629340 us; error 71 us; skew 500 ppm
I20260928 03:04:32.629846 9854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:32 dist-test-slave-fjww krb5kdc[9841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564672, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:32.633632 9854 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:04:32.635294 9854 webserver.cc:460] Webserver started at http://127.6.192.62:36483/ using document root <none> and password file <none>
I20260928 03:04:32.636068 9854 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:32.636188 9854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:32.636479 9854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:32.639901 9854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "b567f821d6e04c91befcd722c172a2e0"
format_stamp: "Formatted at 2026-09-28 03:04:32 on dist-test-slave-fjww"
server_key: "bd91ae6a4cc9c285aab37873d3b1e9f0"
server_key_iv: "36c9e2f27289629c48a5756f6866128e"
server_key_version: "encryptionkey@0"
I20260928 03:04:32.640731 9854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "b567f821d6e04c91befcd722c172a2e0"
format_stamp: "Formatted at 2026-09-28 03:04:32 on dist-test-slave-fjww"
server_key: "bd91ae6a4cc9c285aab37873d3b1e9f0"
server_key_iv: "36c9e2f27289629c48a5756f6866128e"
server_key_version: "encryptionkey@0"
I20260928 03:04:32.646589 9854 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 03:04:32.651043 9869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:32.652931 9854 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:04:32.653275 9854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "b567f821d6e04c91befcd722c172a2e0"
format_stamp: "Formatted at 2026-09-28 03:04:32 on dist-test-slave-fjww"
server_key: "bd91ae6a4cc9c285aab37873d3b1e9f0"
server_key_iv: "36c9e2f27289629c48a5756f6866128e"
server_key_version: "encryptionkey@0"
I20260928 03:04:32.653519 9854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:32.669374 9854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:32.674647 9854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:32.675004 9854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:32.696995 9854 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:46721
I20260928 03:04:32.697041 9920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:46721 every 8 connection(s)
I20260928 03:04:32.698594 9854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:04:32.702859 9921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:32.709165 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 9854
I20260928 03:04:32.709417 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:04:32.710342 6912 external_mini_cluster.cc:1428] Setting key 97bb844066e3e8af80995259f99bc3da
I20260928 03:04:32.716138 9921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0: Bootstrap starting.
I20260928 03:04:32.720256 9921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:32.721710 9921 log.cc:824] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:32.725798 9921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0: No bootstrap required, opened a new log
Sep 28 03:04:32 dist-test-slave-fjww krb5kdc[9841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:32.735270 9921 raft_consensus.cc:348] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b567f821d6e04c91befcd722c172a2e0" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46721 } }
I20260928 03:04:32.735766 9921 raft_consensus.cc:374] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:32.735909 9921 raft_consensus.cc:729] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b567f821d6e04c91befcd722c172a2e0, State: Initialized, Role: FOLLOWER
I20260928 03:04:32.736665 9921 consensus_queue.cc:260] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b567f821d6e04c91befcd722c172a2e0" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46721 } }
I20260928 03:04:32.736937 9921 raft_consensus.cc:388] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:04:32.737119 9921 raft_consensus.cc:482] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:04:32.737336 9921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:32.737816 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:32.712800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45078 (local address 127.6.192.62:46721)
0928 03:04:32.713805 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:32.713818 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:32.713872 (+ 54us) server_negotiation.cc:407] Connection header received
0928 03:04:32.715761 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:32.715788 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:32.716234 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:32.716674 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:32.718760 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:32.720260 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:32.722237 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:32.723544 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:32.723689 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:32.729107 (+ 5418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:32.729116 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:32.729134 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:32.729178 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:32.732734 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:32.733710 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:32.733726 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:32.733734 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:32.733968 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:32.734441 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:32.734446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:32.734451 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:32.735095 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:32.735415 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:32.735954 (+ 539us) server_negotiation.cc:299] Negotiation successful
0928 03:04:32.736493 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":491,"threads_started":1}
I20260928 03:04:32.740515 9921 raft_consensus.cc:504] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b567f821d6e04c91befcd722c172a2e0" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46721 } }
I20260928 03:04:32.741088 9921 leader_election.cc:302] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b567f821d6e04c91befcd722c172a2e0; no voters:
I20260928 03:04:32.742040 9921 leader_election.cc:288] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:04:32.742493 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:04:32.742588 9926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:32.744560 9921 sys_catalog.cc:564] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:04:32.747587 9926 raft_consensus.cc:686] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0 [term 1 LEADER]: Becoming Leader. State: Replica: b567f821d6e04c91befcd722c172a2e0, State: Running, Role: LEADER
I20260928 03:04:32.748741 9926 consensus_queue.cc:237] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b567f821d6e04c91befcd722c172a2e0" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46721 } }
I20260928 03:04:32.753031 9928 sys_catalog.cc:455] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b567f821d6e04c91befcd722c172a2e0. Latest consensus state: current_term: 1 leader_uuid: "b567f821d6e04c91befcd722c172a2e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b567f821d6e04c91befcd722c172a2e0" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46721 } } }
I20260928 03:04:32.753028 9927 sys_catalog.cc:455] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b567f821d6e04c91befcd722c172a2e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b567f821d6e04c91befcd722c172a2e0" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46721 } } }
I20260928 03:04:32.753509 9928 sys_catalog.cc:458] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:32.753587 9927 sys_catalog.cc:458] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:32.766582 9942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:04:32.772239 9942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:04:32.783874 9942 catalog_manager.cc:1347] Generated new cluster ID: 4b17e0f4af4d4d2293cb4205f0d0bd28
I20260928 03:04:32.784000 9942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:04:32.796373 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:32.786807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.6.192.62:46721)
0928 03:04:32.786988 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:32.786994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:32.787620 (+ 626us) server_negotiation.cc:407] Connection header received
0928 03:04:32.787831 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:32.787837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:32.787914 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:32.788014 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:32.789294 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:32.790395 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:32.792346 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:32.792984 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:32.793106 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:32.794400 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:32.794406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:32.794411 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:04:32.795367 (+ 956us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:32.795487 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:04:32.795529 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:32.795640 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:32.795881 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 03:04:32.795989 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 03:04:32.800508 9942 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:04:32.802582 9942 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:04:32.826036 9942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b567f821d6e04c91befcd722c172a2e0: Generated new TSK 0
I20260928 03:04:32.827054 9942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:04:32.836968 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 9854
I20260928 03:04:32.838397 9916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:32.933823 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 9854
2026-09-28T03:04:32Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2033 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:04:33 dist-test-slave-fjww krb5kdc[9955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:04:33 dist-test-slave-fjww krb5kdc[9955](info): set up 2 sockets
Sep 28 03:04:33 dist-test-slave-fjww krb5kdc[9955](info): commencing operation
krb5kdc: starting...
W20260928 03:04:34.064448 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:04:34 dist-test-slave-fjww krb5kdc[9955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:04:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:04:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:04:34.283985 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:43235
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:43235
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:34.625766 9968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:34.626142 9968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:34.626314 9968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:34.633934 9968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:04:34.634106 9968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:34.634209 9968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:34.634316 9968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:04:34.634392 9968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:04:34.644788 9968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:43235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:43235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:34.646286 9968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:34.647737 9968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:34.660717 9973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:34.660766 9976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:34.661033 9974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:34.661492 9968 server_base.cc:1034] running on GCE node
I20260928 03:04:34.662403 9968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:34.664150 9968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:34.665409 9968 hybrid_clock.cc:648] HybridClock initialized: now 1790564674665367 us; error 59 us; skew 500 ppm
I20260928 03:04:34.665812 9968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:34 dist-test-slave-fjww krb5kdc[9955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564674, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:34.673091 9968 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:04:34.675058 9968 webserver.cc:460] Webserver started at http://127.6.192.62:44595/ using document root <none> and password file <none>
I20260928 03:04:34.675926 9968 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:34.676086 9968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:34.676451 9968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:34.680498 9968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "e6d74632d45840f19231cbd4e9453d23"
format_stamp: "Formatted at 2026-09-28 03:04:34 on dist-test-slave-fjww"
server_key: "b7ed579ba03f36626b34b807373acdb3"
server_key_iv: "795f6d4c9acb474ff3a8fb46c6b0210f"
server_key_version: "encryptionkey@0"
I20260928 03:04:34.681576 9968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "e6d74632d45840f19231cbd4e9453d23"
format_stamp: "Formatted at 2026-09-28 03:04:34 on dist-test-slave-fjww"
server_key: "b7ed579ba03f36626b34b807373acdb3"
server_key_iv: "795f6d4c9acb474ff3a8fb46c6b0210f"
server_key_version: "encryptionkey@0"
I20260928 03:04:34.688815 9968 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260928 03:04:34.693285 9983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:34.695209 9968 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 03:04:34.695575 9968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "e6d74632d45840f19231cbd4e9453d23"
format_stamp: "Formatted at 2026-09-28 03:04:34 on dist-test-slave-fjww"
server_key: "b7ed579ba03f36626b34b807373acdb3"
server_key_iv: "795f6d4c9acb474ff3a8fb46c6b0210f"
server_key_version: "encryptionkey@0"
I20260928 03:04:34.695822 9968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:34.711079 9968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:34.716714 9968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:34.717062 9968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:34.739226 9968 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:43235
I20260928 03:04:34.739331 10034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:43235 every 8 connection(s)
I20260928 03:04:34.740820 9968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:04:34.745174 10035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:34.746410 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 9968
I20260928 03:04:34.746665 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:04:34.747622 6912 external_mini_cluster.cc:1428] Setting key 9dc77db18a151c48411e922d1d10e799
I20260928 03:04:34.761765 10035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23: Bootstrap starting.
Sep 28 03:04:34 dist-test-slave-fjww krb5kdc[9955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:34.765925 10035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:34.767364 10035 log.cc:824] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:34.771118 10035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23: No bootstrap required, opened a new log
I20260928 03:04:34.774915 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:34.749940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51310 (local address 127.6.192.62:43235)
0928 03:04:34.750944 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:34.750957 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:34.751001 (+ 44us) server_negotiation.cc:407] Connection header received
0928 03:04:34.752862 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:34.752888 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:34.753341 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:34.753719 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:34.755642 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:34.757178 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:34.759390 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:34.760293 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:34.760452 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:34.765936 (+ 5484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:34.765948 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:34.765967 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:34.766021 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:34.770054 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:34.770939 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:34.770948 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:34.770961 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:34.771156 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:34.771721 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:34.771729 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:34.771735 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:34.772364 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:34.772715 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:34.773260 (+ 545us) server_negotiation.cc:299] Negotiation successful
0928 03:04:34.773761 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":509,"threads_started":1}
I20260928 03:04:34.779954 10035 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d74632d45840f19231cbd4e9453d23" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43235 } }
I20260928 03:04:34.780400 10035 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:34.780521 10035 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6d74632d45840f19231cbd4e9453d23, State: Initialized, Role: FOLLOWER
I20260928 03:04:34.781157 10035 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d74632d45840f19231cbd4e9453d23" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43235 } }
I20260928 03:04:34.781418 10035 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:04:34.781575 10035 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:04:34.781757 10035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:34.784683 10035 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d74632d45840f19231cbd4e9453d23" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43235 } }
I20260928 03:04:34.785234 10035 leader_election.cc:302] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6d74632d45840f19231cbd4e9453d23; no voters:
I20260928 03:04:34.785878 10035 leader_election.cc:288] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:04:34.786057 10040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:34.786866 10040 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23 [term 1 LEADER]: Becoming Leader. State: Replica: e6d74632d45840f19231cbd4e9453d23, State: Running, Role: LEADER
I20260928 03:04:34.787669 10040 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d74632d45840f19231cbd4e9453d23" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43235 } }
I20260928 03:04:34.788149 10035 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:04:34.791204 10042 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6d74632d45840f19231cbd4e9453d23. Latest consensus state: current_term: 1 leader_uuid: "e6d74632d45840f19231cbd4e9453d23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d74632d45840f19231cbd4e9453d23" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43235 } } }
I20260928 03:04:34.791230 10041 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6d74632d45840f19231cbd4e9453d23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d74632d45840f19231cbd4e9453d23" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43235 } } }
I20260928 03:04:34.791718 10042 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:34.791790 10041 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:34.792687 10049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:04:34.813628 10049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:04:34.840190 10049 catalog_manager.cc:1347] Generated new cluster ID: cfb93ebef0d4476b8f773f4646ea2756
I20260928 03:04:34.840339 10049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:04:34.866070 10049 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:04:34.868417 10049 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:04:34.884246 10049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6d74632d45840f19231cbd4e9453d23: Generated new TSK 0
I20260928 03:04:34.885548 10049 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
I20260928 03:04:34.951773 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:43235
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:35.304401 10061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:35.304708 10061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:35.304867 10061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:35.312700 10061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:35.312837 10061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:35.313019 10061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:04:35.323732 10061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:43235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:35.325135 10061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:35.326751 10061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:35.341408 10066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:35.341521 10067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:35.342131 10069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:35.342769 10061 server_base.cc:1034] running on GCE node
I20260928 03:04:35.343637 10061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:35.344719 10061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:35.346104 10061 hybrid_clock.cc:648] HybridClock initialized: now 1790564675346045 us; error 51 us; skew 500 ppm
I20260928 03:04:35.346553 10061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:35 dist-test-slave-fjww krb5kdc[9955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564675, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:35.350407 10061 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:04:35.352185 10061 webserver.cc:460] Webserver started at http://127.6.192.1:39705/ using document root <none> and password file <none>
I20260928 03:04:35.352965 10061 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:35.353106 10061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:35.353451 10061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:35.356817 10061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "2218fb3a42ab4988b1f6722d642185b5"
format_stamp: "Formatted at 2026-09-28 03:04:35 on dist-test-slave-fjww"
server_key: "9ad610058b734a072eeecd68b77413c4"
server_key_iv: "9d4d8aeb1c35c318324b63c6adbe557c"
server_key_version: "encryptionkey@0"
I20260928 03:04:35.357712 10061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "2218fb3a42ab4988b1f6722d642185b5"
format_stamp: "Formatted at 2026-09-28 03:04:35 on dist-test-slave-fjww"
server_key: "9ad610058b734a072eeecd68b77413c4"
server_key_iv: "9d4d8aeb1c35c318324b63c6adbe557c"
server_key_version: "encryptionkey@0"
I20260928 03:04:35.363374 10061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 03:04:35.367667 10076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:35.369307 10061 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:04:35.369541 10061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "2218fb3a42ab4988b1f6722d642185b5"
format_stamp: "Formatted at 2026-09-28 03:04:35 on dist-test-slave-fjww"
server_key: "9ad610058b734a072eeecd68b77413c4"
server_key_iv: "9d4d8aeb1c35c318324b63c6adbe557c"
server_key_version: "encryptionkey@0"
I20260928 03:04:35.369782 10061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:35.385144 10061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:35.390015 10061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:35.390328 10061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:35.391390 10061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:35.392927 10061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:35.393035 10061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:35.393184 10061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:35.393272 10061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:35.429699 10061 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:46453
I20260928 03:04:35.429699 10188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:46453 every 8 connection(s)
I20260928 03:04:35.431149 10061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:04:35.433250 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 10061
I20260928 03:04:35.433457 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:04:35.434305 6912 external_mini_cluster.cc:1428] Setting key b0fc3a2fa159602d04c4e7429d5e39ee
Sep 28 03:04:35 dist-test-slave-fjww krb5kdc[9955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564675, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.192.2@KRBTEST.COM; defaulting to no policy
I20260928 03:04:35.454813 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:35.434355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:54095 (local address 127.6.192.62:43235)
0928 03:04:35.434855 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:35.434859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:35.435973 (+ 1114us) server_negotiation.cc:407] Connection header received
0928 03:04:35.437880 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:35.437885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:35.437974 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:35.438101 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:35.439895 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:35.440633 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:35.443284 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:35.443817 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:35.443947 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:35.448546 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:35.448552 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:35.448556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:35.448595 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:35.451428 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:35.452356 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:35.452363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:35.452371 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:35.452500 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:35.452979 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:35.452989 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:35.452992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:35.453271 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:35.453437 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:35.454237 (+ 800us) server_negotiation.cc:299] Negotiation successful
0928 03:04:35.454421 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":257,"threads_started":1}
I20260928 03:04:35.456040 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:35.435010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43235 (local address 127.6.192.1:54095)
0928 03:04:35.435821 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:35.435868 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:35.437591 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:35.438397 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:35.438405 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:35.438994 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:35.439695 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:35.439724 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:35.440824 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:35.440828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:35.443073 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:35.443094 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:35.444088 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:35.444098 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:35.444443 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:35.445290 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:35.445326 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:35.448225 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:35.451688 (+ 3463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:35.451699 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:35.451726 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:35.452201 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:35.452622 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:35.452627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:35.452631 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:35.452847 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:35.453447 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:35.453459 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:35.453886 (+ 427us) client_negotiation.cc:769] Sending connection context
0928 03:04:35.454129 (+ 243us) client_negotiation.cc:240] Negotiation successful
0928 03:04:35.454626 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":318,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.2@KRBTEST.COM" created.
I20260928 03:04:35.457993 10189 heartbeater.cc:347] Connected to a master server at 127.6.192.62:43235
I20260928 03:04:35.458405 10189 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:35.459370 10189 heartbeater.cc:511] Master 127.6.192.62:43235 requested a full tablet report, sending...
I20260928 03:04:35.462018 10000 ts_manager.cc:194] Registered new tserver with Master: 2218fb3a42ab4988b1f6722d642185b5 (127.6.192.1:46453)
I20260928 03:04:35.464689 10000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:54095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
I20260928 03:04:35.487425 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:43235
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:35.837713 10195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:35.838019 10195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:35.838119 10195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:35.845616 10195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:35.845755 10195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:35.845871 10195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:04:35.856020 10195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:43235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:35.857715 10195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:35.859119 10195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:35.874267 10201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:35.874294 10203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:35.874697 10200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:35.875178 10195 server_base.cc:1034] running on GCE node
I20260928 03:04:35.875941 10195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:35.876940 10195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:35.878192 10195 hybrid_clock.cc:648] HybridClock initialized: now 1790564675878127 us; error 65 us; skew 500 ppm
I20260928 03:04:35.878556 10195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:35 dist-test-slave-fjww krb5kdc[9955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564675, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:35.882000 10195 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:04:35.883800 10195 webserver.cc:460] Webserver started at http://127.6.192.2:34715/ using document root <none> and password file <none>
I20260928 03:04:35.884692 10195 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:35.884848 10195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:35.885177 10195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:35.888386 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "c3439d953c17466693069b153c393b40"
format_stamp: "Formatted at 2026-09-28 03:04:35 on dist-test-slave-fjww"
server_key: "80234b599c5366124941d75ea8d1c2c2"
server_key_iv: "0fcca426f39339cb7eb410ea80e866fa"
server_key_version: "encryptionkey@0"
I20260928 03:04:35.889303 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "c3439d953c17466693069b153c393b40"
format_stamp: "Formatted at 2026-09-28 03:04:35 on dist-test-slave-fjww"
server_key: "80234b599c5366124941d75ea8d1c2c2"
server_key_iv: "0fcca426f39339cb7eb410ea80e866fa"
server_key_version: "encryptionkey@0"
I20260928 03:04:35.895349 10195 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 03:04:35.899590 10210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:35.901592 10195 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:04:35.901810 10195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "c3439d953c17466693069b153c393b40"
format_stamp: "Formatted at 2026-09-28 03:04:35 on dist-test-slave-fjww"
server_key: "80234b599c5366124941d75ea8d1c2c2"
server_key_iv: "0fcca426f39339cb7eb410ea80e866fa"
server_key_version: "encryptionkey@0"
I20260928 03:04:35.902050 10195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:35.932448 10195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:35.937799 10195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:35.938162 10195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:35.939492 10195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:35.940984 10195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:35.941097 10195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:35.941280 10195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:35.941356 10195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:35.978251 10195 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:35965
I20260928 03:04:35.978345 10322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:35965 every 8 connection(s)
I20260928 03:04:35.979745 10195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:04:35.989691 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 10195
I20260928 03:04:35.989895 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:04:35.990392 6912 external_mini_cluster.cc:1428] Setting key aa096173b6794c38636bfd7482fbe8e8
Sep 28 03:04:35 dist-test-slave-fjww krb5kdc[9955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564675, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:36.005470 10325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:35.982800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:46767 (local address 127.6.192.62:43235)
0928 03:04:35.983294 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:35.983298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:35.984519 (+ 1221us) server_negotiation.cc:407] Connection header received
0928 03:04:35.986608 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:35.986614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:35.986729 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:35.986864 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:35.988757 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:35.989615 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:35.991847 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:35.992336 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:35.992506 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:35.996954 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:35.996960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:35.996963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:35.997003 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:36.001022 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:36.002192 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:36.002202 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:36.002209 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:36.002413 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:36.002976 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:36.002986 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:36.002993 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:36.003481 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:36.003670 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:36.004580 (+ 910us) server_negotiation.cc:299] Negotiation successful
0928 03:04:36.004878 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":232,"threads_started":1}
I20260928 03:04:36.006189 10326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:35.983369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43235 (local address 127.6.192.2:46767)
0928 03:04:35.984317 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:35.984370 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:35.986291 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:35.987143 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:35.987152 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:35.987839 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:35.988527 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:35.988575 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:35.989837 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:35.989845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:35.991654 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:35.991682 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:35.992653 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:35.992662 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:35.992981 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:35.993771 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:35.993797 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:35.996623 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:36.001351 (+ 4728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:36.001360 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:36.001377 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:36.001996 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:36.002558 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:36.002562 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:36.002565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:36.002817 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:36.003649 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:36.003657 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:36.004036 (+ 379us) client_negotiation.cc:769] Sending connection context
0928 03:04:36.004391 (+ 355us) client_negotiation.cc:240] Negotiation successful
0928 03:04:36.004854 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":327,"threads_started":1}
I20260928 03:04:36.008109 10323 heartbeater.cc:347] Connected to a master server at 127.6.192.62:43235
I20260928 03:04:36.008527 10323 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:36.009686 10323 heartbeater.cc:511] Master 127.6.192.62:43235 requested a full tablet report, sending...
I20260928 03:04:36.011963 10000 ts_manager.cc:194] Registered new tserver with Master: c3439d953c17466693069b153c393b40 (127.6.192.2:35965)
WARNING: no policy specified for kudu/127.6.192.3@KRBTEST.COM; defaulting to no policy
I20260928 03:04:36.013162 10000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:46767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
I20260928 03:04:36.044471 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:43235
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:36.384047 10329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:36.384356 10329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:36.384454 10329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:36.392208 10329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:36.392343 10329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:36.392462 10329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:04:36.402743 10329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:43235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:36.404131 10329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:36.405408 10329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:36.420087 10334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:36.420166 10335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:36.420087 10337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:36.421476 10329 server_base.cc:1034] running on GCE node
I20260928 03:04:36.422204 10329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:36.423322 10329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:36.424567 10329 hybrid_clock.cc:648] HybridClock initialized: now 1790564676424510 us; error 52 us; skew 500 ppm
I20260928 03:04:36.424949 10329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:36 dist-test-slave-fjww krb5kdc[9955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564676, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:36.428889 10329 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:04:36.431207 10329 webserver.cc:460] Webserver started at http://127.6.192.3:41315/ using document root <none> and password file <none>
I20260928 03:04:36.432215 10329 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:36.432368 10329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:36.432723 10329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:36.436746 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "43d2899e42c946baa897116d35cd6d1b"
format_stamp: "Formatted at 2026-09-28 03:04:36 on dist-test-slave-fjww"
server_key: "a009bc586f2d759163409b4a1b9da957"
server_key_iv: "6326ce6b4ef8ab9556ea0d4832aa4f77"
server_key_version: "encryptionkey@0"
I20260928 03:04:36.437600 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "43d2899e42c946baa897116d35cd6d1b"
format_stamp: "Formatted at 2026-09-28 03:04:36 on dist-test-slave-fjww"
server_key: "a009bc586f2d759163409b4a1b9da957"
server_key_iv: "6326ce6b4ef8ab9556ea0d4832aa4f77"
server_key_version: "encryptionkey@0"
I20260928 03:04:36.443286 10329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 03:04:36.447501 10344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:36.449126 10329 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:04:36.449298 10329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "43d2899e42c946baa897116d35cd6d1b"
format_stamp: "Formatted at 2026-09-28 03:04:36 on dist-test-slave-fjww"
server_key: "a009bc586f2d759163409b4a1b9da957"
server_key_iv: "6326ce6b4ef8ab9556ea0d4832aa4f77"
server_key_version: "encryptionkey@0"
I20260928 03:04:36.449579 10329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:36.469476 10189 heartbeater.cc:503] Master 127.6.192.62:43235 was elected leader, sending a full tablet report...
I20260928 03:04:36.494304 10329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:36.499699 10329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:36.500082 10329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:36.501456 10329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:36.503031 10329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:36.503155 10329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:36.503357 10329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:36.503444 10329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:36.542898 10329 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:34563
I20260928 03:04:36.542950 10456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:34563 every 8 connection(s)
I20260928 03:04:36.544387 10329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:04:36.547035 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 10329
I20260928 03:04:36.547240 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:04:36.547896 6912 external_mini_cluster.cc:1428] Setting key 8a23967245075fbb496ab16031b7837d
Sep 28 03:04:36 dist-test-slave-fjww krb5kdc[9955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564676, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:36.568284 10459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:36.547224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:33953 (local address 127.6.192.62:43235)
0928 03:04:36.547798 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:36.547805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:36.549024 (+ 1219us) server_negotiation.cc:407] Connection header received
0928 03:04:36.551511 (+ 2487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:36.551517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:36.551612 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:36.551708 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:36.553450 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:36.554516 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:36.556257 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:36.556685 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:36.556797 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:36.560824 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:36.560830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:36.560834 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:36.560871 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:36.564615 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:36.565552 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:36.565557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:36.565560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:36.565691 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:36.566266 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:36.566270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:36.566273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:36.566568 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:36.566705 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:36.567678 (+ 973us) server_negotiation.cc:299] Negotiation successful
0928 03:04:36.567885 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":285,"threads_started":1}
I20260928 03:04:36.569350 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:36.547940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43235 (local address 127.6.192.3:33953)
0928 03:04:36.548827 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:36.548882 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:36.551152 (+ 2270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:36.551932 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:36.551941 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:36.552585 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:36.553225 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:36.553243 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:36.554672 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:36.554675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:36.556083 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:36.556095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:36.556903 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:36.556908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:36.557113 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:36.557868 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:36.557894 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:36.560524 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:36.564831 (+ 4307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:36.564841 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:36.564857 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:36.565382 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:36.565813 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:36.565819 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:36.565823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:36.566038 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:36.566726 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:36.566733 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:36.567156 (+ 423us) client_negotiation.cc:769] Sending connection context
0928 03:04:36.567514 (+ 358us) client_negotiation.cc:240] Negotiation successful
0928 03:04:36.568028 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":377,"threads_started":1}
I20260928 03:04:36.571066 10457 heartbeater.cc:347] Connected to a master server at 127.6.192.62:43235
I20260928 03:04:36.571378 10457 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:36.572269 10457 heartbeater.cc:511] Master 127.6.192.62:43235 requested a full tablet report, sending...
I20260928 03:04:36.573937 10000 ts_manager.cc:194] Registered new tserver with Master: 43d2899e42c946baa897116d35cd6d1b (127.6.192.3:34563)
I20260928 03:04:36.574872 10000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:33953
I20260928 03:04:36.577551 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 03:04:36 dist-test-slave-fjww krb5kdc[9955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564676, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 03:04:36 dist-test-slave-fjww krb5kdc[9955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564676, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:36.633409 10459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:36.616468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.6.192.62:43235)
0928 03:04:36.616656 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:36.616661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:36.617114 (+ 453us) server_negotiation.cc:407] Connection header received
0928 03:04:36.617290 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:36.617295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:36.617374 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:36.617486 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:36.619027 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:36.620039 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:36.621537 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:36.621983 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:36.622107 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:36.626837 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:36.626843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:36.626847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:36.626892 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:36.629954 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:36.630970 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:36.630977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:36.630985 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:36.631133 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:36.631643 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:36.631650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:36.631654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:36.632067 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:36.632354 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:36.632713 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 03:04:36.633047 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 03:04:36.634019 10000 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51326
I20260928 03:04:36.636412 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 10061
Sep 28 03:04:36 dist-test-slave-fjww krb5kdc[9955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564676, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:04:36.654903 10469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:36.637468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.6.192.1:46453)
0928 03:04:36.638367 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:36.638375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:36.638400 (+ 25us) server_negotiation.cc:407] Connection header received
0928 03:04:36.638471 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:36.638478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:36.638712 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:36.638851 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:36.640290 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:36.641553 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:36.642821 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:36.643632 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:36.643745 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:36.648088 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:36.648119 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:36.648136 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:36.648281 (+ 145us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:36.651660 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:36.652502 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:36.652510 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:36.652512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:36.652665 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:36.653068 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:36.653072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:36.653074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:36.653566 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:36.653905 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:36.654186 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 03:04:36.654532 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":573,"threads_started":1}
W20260928 03:04:36.656454 10184 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36484
W20260928 03:04:36.658064 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 10061: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:04:36.658322 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 10061
I20260928 03:04:36.675881 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 10195
Sep 28 03:04:36 dist-test-slave-fjww krb5kdc[9955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564676, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.192.2@KRBTEST.COM
I20260928 03:04:36.694367 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:36.676848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.6.192.2:35965)
0928 03:04:36.677445 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:36.677453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:36.677481 (+ 28us) server_negotiation.cc:407] Connection header received
0928 03:04:36.677545 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:36.677552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:36.677771 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:36.677923 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:36.679167 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:36.680398 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:36.681723 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:36.682350 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:36.682483 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:36.687284 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:36.687305 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:36.687325 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:36.687387 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:36.690771 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:36.691678 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:36.691687 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:36.691689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:36.691887 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:36.692314 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:36.692320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:36.692322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:36.692756 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:36.693094 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:36.693596 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 03:04:36.693882 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":308,"threads_started":1}
W20260928 03:04:36.695909 10318 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58800
W20260928 03:04:36.697166 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 10195: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:04:36.697366 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 10195
I20260928 03:04:36.714159 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 10329
Sep 28 03:04:36 dist-test-slave-fjww krb5kdc[9955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564676, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.192.3@KRBTEST.COM
I20260928 03:04:36.733824 10471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:36.715257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59904 (local address 127.6.192.3:34563)
0928 03:04:36.716133 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:36.716141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:36.716174 (+ 33us) server_negotiation.cc:407] Connection header received
0928 03:04:36.716238 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:36.716244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:36.716479 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:36.716627 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:36.718130 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:36.719371 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:36.720859 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:36.721593 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:36.721711 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:36.727016 (+ 5305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:36.727031 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:36.727047 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:36.727092 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:36.730494 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:36.731364 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:36.731382 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:36.731385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:36.731649 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:36.732086 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:36.732090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:36.732093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:36.732586 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:36.732864 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:36.733162 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 03:04:36.733503 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":512,"threads_started":1}
W20260928 03:04:36.735396 10452 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59904
W20260928 03:04:36.736974 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 10329: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:04:36.737157 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 10329
I20260928 03:04:36.754817 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 9968
I20260928 03:04:36.756209 10030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:36.866892 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 9968
2026-09-28T03:04:36Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3936 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:04:36 dist-test-slave-fjww krb5kdc[10475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:04:36 dist-test-slave-fjww krb5kdc[10475](info): set up 2 sockets
Sep 28 03:04:36 dist-test-slave-fjww krb5kdc[10475](info): commencing operation
krb5kdc: starting...
W20260928 03:04:38.008157 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:04:38 dist-test-slave-fjww krb5kdc[10475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:04:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:04:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:04:38.229549 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:41761
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:40283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:41761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:38.571839 10488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:38.572176 10488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:38.572384 10488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:38.580235 10488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:04:38.580367 10488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:38.580461 10488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:38.580555 10488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:04:38.580662 10488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:04:38.591095 10488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:40283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:41761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:41761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:38.592581 10488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:38.594072 10488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:38.607127 10493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:38.607046 10496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:38.607146 10494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:38.608433 10488 server_base.cc:1034] running on GCE node
I20260928 03:04:38.609424 10488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:38.610985 10488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:38.612247 10488 hybrid_clock.cc:648] HybridClock initialized: now 1790564678612168 us; error 71 us; skew 500 ppm
I20260928 03:04:38.612603 10488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:38 dist-test-slave-fjww krb5kdc[10475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564678, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:38.616386 10488 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:04:38.618105 10488 webserver.cc:460] Webserver started at http://127.6.192.62:40411/ using document root <none> and password file <none>
I20260928 03:04:38.618948 10488 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:38.619135 10488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:38.619558 10488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:38.622829 10488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "56d3180bfa244b2fbcb92dd7421c5354"
format_stamp: "Formatted at 2026-09-28 03:04:38 on dist-test-slave-fjww"
server_key: "44d12e24777b180f04d0365fc95d5b73"
server_key_iv: "281b278b9046bcd7d88cc542c9c4ce26"
server_key_version: "encryptionkey@0"
I20260928 03:04:38.623798 10488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "56d3180bfa244b2fbcb92dd7421c5354"
format_stamp: "Formatted at 2026-09-28 03:04:38 on dist-test-slave-fjww"
server_key: "44d12e24777b180f04d0365fc95d5b73"
server_key_iv: "281b278b9046bcd7d88cc542c9c4ce26"
server_key_version: "encryptionkey@0"
I20260928 03:04:38.629684 10488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:04:38.633750 10503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:38.635540 10488 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260928 03:04:38.635759 10488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "56d3180bfa244b2fbcb92dd7421c5354"
format_stamp: "Formatted at 2026-09-28 03:04:38 on dist-test-slave-fjww"
server_key: "44d12e24777b180f04d0365fc95d5b73"
server_key_iv: "281b278b9046bcd7d88cc542c9c4ce26"
server_key_version: "encryptionkey@0"
I20260928 03:04:38.636013 10488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:38.657773 10488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:38.662448 10488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:38.662698 10488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:38.682703 10488 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:41761
I20260928 03:04:38.682741 10554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:41761 every 8 connection(s)
I20260928 03:04:38.684290 10488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:04:38.688802 10555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:38.690740 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 10488
I20260928 03:04:38.691000 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:04:38.691715 6912 external_mini_cluster.cc:1428] Setting key 6efb040e5d5132252efa1c75e3777159
I20260928 03:04:38.701057 10555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354: Bootstrap starting.
I20260928 03:04:38.704346 10555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:38.705456 10555 log.cc:824] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354: Log is configured to *not* fsync() on all Append() calls
Sep 28 03:04:38 dist-test-slave-fjww krb5kdc[10475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:38.708932 10555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354: No bootstrap required, opened a new log
I20260928 03:04:38.716014 10558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:38.693873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.6.192.62:41761)
0928 03:04:38.694826 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:38.694842 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:38.694929 (+ 87us) server_negotiation.cc:407] Connection header received
0928 03:04:38.696700 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:38.696726 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:38.697154 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:38.697536 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:38.699102 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:38.700538 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:38.701888 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:38.702596 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:38.702819 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:38.707164 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:38.707172 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:38.707189 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:38.707238 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:38.710711 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:38.711732 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:38.711739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:38.711747 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:38.711933 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:38.712567 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:38.712572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:38.712574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:38.713289 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:38.713560 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:38.714155 (+ 595us) server_negotiation.cc:299] Negotiation successful
0928 03:04:38.714750 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":358,"threads_started":1}
I20260928 03:04:38.718292 10555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d3180bfa244b2fbcb92dd7421c5354" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41761 } }
I20260928 03:04:38.718737 10555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:38.718863 10555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d3180bfa244b2fbcb92dd7421c5354, State: Initialized, Role: FOLLOWER
I20260928 03:04:38.719533 10555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d3180bfa244b2fbcb92dd7421c5354" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41761 } }
I20260928 03:04:38.719761 10555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:04:38.719861 10555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:04:38.719978 10555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:38.722134 10555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d3180bfa244b2fbcb92dd7421c5354" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41761 } }
I20260928 03:04:38.722611 10555 leader_election.cc:302] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56d3180bfa244b2fbcb92dd7421c5354; no voters:
I20260928 03:04:38.723170 10555 leader_election.cc:288] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:04:38.723417 10560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:38.724169 10560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354 [term 1 LEADER]: Becoming Leader. State: Replica: 56d3180bfa244b2fbcb92dd7421c5354, State: Running, Role: LEADER
I20260928 03:04:38.724853 10560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d3180bfa244b2fbcb92dd7421c5354" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41761 } }
I20260928 03:04:38.725457 10555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:04:38.728576 10562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56d3180bfa244b2fbcb92dd7421c5354. Latest consensus state: current_term: 1 leader_uuid: "56d3180bfa244b2fbcb92dd7421c5354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d3180bfa244b2fbcb92dd7421c5354" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41761 } } }
I20260928 03:04:38.728544 10561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56d3180bfa244b2fbcb92dd7421c5354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d3180bfa244b2fbcb92dd7421c5354" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41761 } } }
I20260928 03:04:38.728988 10562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:38.729027 10561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:38.731019 10567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:04:38.738155 10567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:04:38.749913 10567 catalog_manager.cc:1347] Generated new cluster ID: 7e24ab1a9bbb4c97b3dd0d5b8fe7da65
I20260928 03:04:38.750118 10567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:04:38.772524 10567 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:04:38.774230 10567 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:04:38.790484 10567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56d3180bfa244b2fbcb92dd7421c5354: Generated new TSK 0
I20260928 03:04:38.791719 10567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
I20260928 03:04:38.856230 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41761
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:40283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:39.202180 10581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:39.202469 10581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:39.202630 10581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:39.210078 10581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:39.210203 10581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:39.210422 10581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:04:39.220866 10581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:40283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:39.222286 10581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:39.223755 10581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:39.238214 10587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:39.238251 10586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:39.238442 10589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:39.239184 10581 server_base.cc:1034] running on GCE node
I20260928 03:04:39.239988 10581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:39.241217 10581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:39.242449 10581 hybrid_clock.cc:648] HybridClock initialized: now 1790564679242397 us; error 38 us; skew 500 ppm
I20260928 03:04:39.242833 10581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:39 dist-test-slave-fjww krb5kdc[10475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564679, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:39.246390 10581 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:04:39.248117 10581 webserver.cc:460] Webserver started at http://127.6.192.1:38231/ using document root <none> and password file <none>
I20260928 03:04:39.248872 10581 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:39.249038 10581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:39.249372 10581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:39.252630 10581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "9a80efd5cec54a529118de43427488e4"
format_stamp: "Formatted at 2026-09-28 03:04:39 on dist-test-slave-fjww"
server_key: "51a1b98791538c64420e51ab0467e973"
server_key_iv: "5b7722d9fd2fc3da6468484076cc2d94"
server_key_version: "encryptionkey@0"
I20260928 03:04:39.253504 10581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "9a80efd5cec54a529118de43427488e4"
format_stamp: "Formatted at 2026-09-28 03:04:39 on dist-test-slave-fjww"
server_key: "51a1b98791538c64420e51ab0467e973"
server_key_iv: "5b7722d9fd2fc3da6468484076cc2d94"
server_key_version: "encryptionkey@0"
I20260928 03:04:39.259737 10581 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20260928 03:04:39.263971 10596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:39.265671 10581 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 03:04:39.265884 10581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "9a80efd5cec54a529118de43427488e4"
format_stamp: "Formatted at 2026-09-28 03:04:39 on dist-test-slave-fjww"
server_key: "51a1b98791538c64420e51ab0467e973"
server_key_iv: "5b7722d9fd2fc3da6468484076cc2d94"
server_key_version: "encryptionkey@0"
I20260928 03:04:39.266124 10581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:39.280745 10581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:39.285672 10581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:39.285995 10581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:39.287083 10581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:39.288589 10581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:39.288703 10581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:39.288857 10581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:39.288937 10581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:39.325033 10581 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:37877
I20260928 03:04:39.325078 10708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:37877 every 8 connection(s)
I20260928 03:04:39.326545 10581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:04:39.327943 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 10581
I20260928 03:04:39.328174 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:04:39.329017 6912 external_mini_cluster.cc:1428] Setting key 7b8b93adbb79a64e68247b812e4dc359
Sep 28 03:04:39 dist-test-slave-fjww krb5kdc[10475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564679, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.192.2@KRBTEST.COM; defaulting to no policy
I20260928 03:04:39.349359 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:39.329839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:45507 (local address 127.6.192.62:41761)
0928 03:04:39.330299 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:39.330303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:39.331510 (+ 1207us) server_negotiation.cc:407] Connection header received
0928 03:04:39.333475 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:39.333481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:39.333578 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:39.333722 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:39.335348 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:39.336213 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:39.337987 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:39.338424 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:39.338591 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:39.342701 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:39.342707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:39.342711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:39.342749 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:39.345755 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:39.346664 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:39.346670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:39.346673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:39.346835 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:39.347323 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:39.347329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:39.347332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:39.347711 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:39.347882 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:39.348701 (+ 819us) server_negotiation.cc:299] Negotiation successful
0928 03:04:39.348981 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":230,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.2@KRBTEST.COM" created.
I20260928 03:04:39.350663 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:39.330450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41761 (local address 127.6.192.1:45507)
0928 03:04:39.331336 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:39.331382 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:39.333142 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:39.333981 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:39.333990 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:39.334546 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:39.335150 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:39.335168 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:39.336398 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:39.336402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:39.337824 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:39.337839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:39.338693 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:39.338701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:39.338964 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:39.339802 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:39.339827 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:39.342403 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:39.345976 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:39.345987 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:39.346012 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:39.346517 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:39.346971 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:39.346976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:39.346980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:39.347199 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:39.347851 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:39.347861 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:39.348251 (+ 390us) client_negotiation.cc:769] Sending connection context
0928 03:04:39.348588 (+ 337us) client_negotiation.cc:240] Negotiation successful
0928 03:04:39.349195 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":383,"threads_started":1}
I20260928 03:04:39.352752 10709 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41761
I20260928 03:04:39.353154 10709 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:39.354080 10709 heartbeater.cc:511] Master 127.6.192.62:41761 requested a full tablet report, sending...
I20260928 03:04:39.356922 10520 ts_manager.cc:194] Registered new tserver with Master: 9a80efd5cec54a529118de43427488e4 (127.6.192.1:37877)
I20260928 03:04:39.358712 10520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:45507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
I20260928 03:04:39.379910 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41761
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:40283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:39.723114 10715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:39.723435 10715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:39.723694 10715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:39.731760 10715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:39.731894 10715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:39.732081 10715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:04:39.742822 10715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:40283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:39.744284 10715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:39.745714 10715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:39.760979 10720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:39.761034 10723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:39.760972 10721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:39.762437 10715 server_base.cc:1034] running on GCE node
I20260928 03:04:39.763155 10715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:39.764278 10715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:39.765530 10715 hybrid_clock.cc:648] HybridClock initialized: now 1790564679765453 us; error 60 us; skew 500 ppm
I20260928 03:04:39.765996 10715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:39 dist-test-slave-fjww krb5kdc[10475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564679, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:39.773111 10715 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:04:39.774941 10715 webserver.cc:460] Webserver started at http://127.6.192.2:38837/ using document root <none> and password file <none>
I20260928 03:04:39.775882 10715 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:39.776050 10715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:39.776400 10715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:39.779721 10715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "e4c888719e25416b993ef8835b1e81df"
format_stamp: "Formatted at 2026-09-28 03:04:39 on dist-test-slave-fjww"
server_key: "92eabf3fb22210c0c7b7d849e699569b"
server_key_iv: "098d5d922790249ef0e105e10fc753a8"
server_key_version: "encryptionkey@0"
I20260928 03:04:39.780624 10715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "e4c888719e25416b993ef8835b1e81df"
format_stamp: "Formatted at 2026-09-28 03:04:39 on dist-test-slave-fjww"
server_key: "92eabf3fb22210c0c7b7d849e699569b"
server_key_iv: "098d5d922790249ef0e105e10fc753a8"
server_key_version: "encryptionkey@0"
I20260928 03:04:39.786710 10715 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 03:04:39.790979 10730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:39.792774 10715 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:04:39.793009 10715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "e4c888719e25416b993ef8835b1e81df"
format_stamp: "Formatted at 2026-09-28 03:04:39 on dist-test-slave-fjww"
server_key: "92eabf3fb22210c0c7b7d849e699569b"
server_key_iv: "098d5d922790249ef0e105e10fc753a8"
server_key_version: "encryptionkey@0"
I20260928 03:04:39.793272 10715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:39.817689 10715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:39.823287 10715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:39.823669 10715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:39.824777 10715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:39.826390 10715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:39.826499 10715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:39.826675 10715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:39.826745 10715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:39.865201 10715 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:40873
I20260928 03:04:39.865262 10842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:40873 every 8 connection(s)
I20260928 03:04:39.866732 10715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:04:39.872159 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 10715
I20260928 03:04:39.872394 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:04:39.873011 6912 external_mini_cluster.cc:1428] Setting key b8c0951598083aeaed9df263ccb37cb1
Sep 28 03:04:39 dist-test-slave-fjww krb5kdc[10475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564679, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:39.894976 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:39.869774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:38777 (local address 127.6.192.62:41761)
0928 03:04:39.870236 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:39.870240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:39.871786 (+ 1546us) server_negotiation.cc:407] Connection header received
0928 03:04:39.874634 (+ 2848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:39.874641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:39.874739 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:39.874896 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:39.877363 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:39.878180 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:39.881297 (+ 3117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:39.881957 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:39.882213 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:39.887777 (+ 5564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:39.887784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:39.887787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:39.887830 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:39.891083 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:39.892226 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:39.892233 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:39.892236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:39.892383 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:39.892927 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:39.892931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:39.892934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:39.893256 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:39.893433 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:39.894254 (+ 821us) server_negotiation.cc:299] Negotiation successful
0928 03:04:39.894541 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":245,"threads_started":1}
WARNING: no policy specified for kudu/127.6.192.3@KRBTEST.COM; defaulting to no policy
I20260928 03:04:39.896189 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:39.870433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41761 (local address 127.6.192.2:38777)
0928 03:04:39.871567 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:39.871622 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:39.874293 (+ 2671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:39.875182 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:39.875193 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:39.876064 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:39.877047 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:39.877081 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:39.878392 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:39.878402 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:39.881031 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:39.881057 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:39.882482 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:39.882491 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:39.882790 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:39.883800 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:39.883852 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:39.887289 (+ 3437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:39.891308 (+ 4019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:39.891325 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:39.891342 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:39.892047 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:39.892534 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:39.892540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:39.892544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:39.892770 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:39.893410 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:39.893420 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:39.893840 (+ 420us) client_negotiation.cc:769] Sending connection context
0928 03:04:39.894147 (+ 307us) client_negotiation.cc:240] Negotiation successful
0928 03:04:39.894786 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":457,"threads_started":1}
I20260928 03:04:39.898409 10843 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41761
I20260928 03:04:39.898773 10843 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.3@KRBTEST.COM" created.
I20260928 03:04:39.899863 10843 heartbeater.cc:511] Master 127.6.192.62:41761 requested a full tablet report, sending...
I20260928 03:04:39.901818 10520 ts_manager.cc:194] Registered new tserver with Master: e4c888719e25416b993ef8835b1e81df (127.6.192.2:40873)
I20260928 03:04:39.903195 10520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:38777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
I20260928 03:04:39.927842 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41761
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:40283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:40.275542 10849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:40.275904 10849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:40.276005 10849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:40.283699 10849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:40.283831 10849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:40.283957 10849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:04:40.294127 10849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:40283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:40.295495 10849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:40.296819 10849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:40.310990 10854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:40.311043 10855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:40.311108 10857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:40.311693 10849 server_base.cc:1034] running on GCE node
I20260928 03:04:40.312355 10849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:40.313426 10849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:40.314677 10849 hybrid_clock.cc:648] HybridClock initialized: now 1790564680314601 us; error 71 us; skew 500 ppm
I20260928 03:04:40.315019 10849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:40 dist-test-slave-fjww krb5kdc[10475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564680, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:40.318502 10849 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:04:40.320330 10849 webserver.cc:460] Webserver started at http://127.6.192.3:35433/ using document root <none> and password file <none>
I20260928 03:04:40.321117 10849 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:40.321277 10849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:40.321609 10849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:40.324954 10849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "cc4910a26dd949a98fc0d144fd1a298d"
format_stamp: "Formatted at 2026-09-28 03:04:40 on dist-test-slave-fjww"
server_key: "f752ed2f4a340c1c9c3dd8d262e1dfdd"
server_key_iv: "3951fda666c3752b4d1d2fef6ae265dd"
server_key_version: "encryptionkey@0"
I20260928 03:04:40.325805 10849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "cc4910a26dd949a98fc0d144fd1a298d"
format_stamp: "Formatted at 2026-09-28 03:04:40 on dist-test-slave-fjww"
server_key: "f752ed2f4a340c1c9c3dd8d262e1dfdd"
server_key_iv: "3951fda666c3752b4d1d2fef6ae265dd"
server_key_version: "encryptionkey@0"
I20260928 03:04:40.331559 10849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 03:04:40.335255 10864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:40.337155 10849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:04:40.337410 10849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "cc4910a26dd949a98fc0d144fd1a298d"
format_stamp: "Formatted at 2026-09-28 03:04:40 on dist-test-slave-fjww"
server_key: "f752ed2f4a340c1c9c3dd8d262e1dfdd"
server_key_iv: "3951fda666c3752b4d1d2fef6ae265dd"
server_key_version: "encryptionkey@0"
I20260928 03:04:40.337673 10849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:40.362303 10709 heartbeater.cc:503] Master 127.6.192.62:41761 was elected leader, sending a full tablet report...
I20260928 03:04:40.363305 10849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:40.368654 10849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:40.369056 10849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:40.370314 10849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:40.372040 10849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:40.372171 10849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:40.372342 10849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:40.372429 10849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:40.411218 10849 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:44371
I20260928 03:04:40.411316 10976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:44371 every 8 connection(s)
I20260928 03:04:40.412623 10849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:04:40.420244 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 10849
I20260928 03:04:40.420501 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:04:40.420928 6912 external_mini_cluster.cc:1428] Setting key dd78c705601e2636b617f2f848cbf5f7
Sep 28 03:04:40 dist-test-slave-fjww krb5kdc[10475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564680, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:40.434643 10979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:40.415677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:46969 (local address 127.6.192.62:41761)
0928 03:04:40.416200 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:40.416207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:40.417005 (+ 798us) server_negotiation.cc:407] Connection header received
0928 03:04:40.419052 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:40.419058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:40.419172 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:40.419299 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:40.421085 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:40.422056 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:40.424015 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:40.424515 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:40.424621 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:40.428735 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:40.428740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:40.428744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:40.428778 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:40.431487 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:40.432339 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:40.432343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:40.432347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:40.432460 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:40.432900 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:40.432903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:40.432905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:40.433164 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:40.433268 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:40.434084 (+ 816us) server_negotiation.cc:299] Negotiation successful
0928 03:04:40.434258 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":279,"threads_started":1}
I20260928 03:04:40.435825 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:40.416086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41761 (local address 127.6.192.3:46969)
0928 03:04:40.416831 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:40.416876 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:40.418757 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:40.419593 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:40.419602 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:40.420154 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:40.420892 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:40.420911 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:40.422232 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:40.422238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:40.423864 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:40.423884 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:40.424739 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:40.424745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:40.424959 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:40.425762 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:40.425791 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:40.428496 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:40.431660 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:40.431669 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:40.431684 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:40.432210 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:40.432559 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:40.432563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:40.432566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:40.432797 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:40.433288 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:40.433294 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:40.433657 (+ 363us) client_negotiation.cc:769] Sending connection context
0928 03:04:40.433955 (+ 298us) client_negotiation.cc:240] Negotiation successful
0928 03:04:40.434391 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":301,"threads_started":1}
I20260928 03:04:40.437395 10977 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41761
I20260928 03:04:40.437767 10977 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:40.438580 10977 heartbeater.cc:511] Master 127.6.192.62:41761 requested a full tablet report, sending...
I20260928 03:04:40.440383 10520 ts_manager.cc:194] Registered new tserver with Master: cc4910a26dd949a98fc0d144fd1a298d (127.6.192.3:44371)
I20260928 03:04:40.441480 10520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:46969
I20260928 03:04:40.449510 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 03:04:40 dist-test-slave-fjww krb5kdc[10475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 03:04:40 dist-test-slave-fjww krb5kdc[10475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:04:40.502703 10985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:40.486615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57986 (local address 127.6.192.1:37877)
0928 03:04:40.487253 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:40.487261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:40.487290 (+ 29us) server_negotiation.cc:407] Connection header received
0928 03:04:40.487359 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:40.487365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:40.487674 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:40.487824 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:40.489257 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:40.490321 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:40.491686 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:40.492327 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:40.492447 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:40.496544 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:40.496560 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:40.496575 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:40.496627 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:40.499797 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:40.500477 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:40.500485 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:40.500487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:40.500644 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:40.501025 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:40.501028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:40.501031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:40.501435 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:40.501689 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:40.502135 (+ 446us) server_negotiation.cc:299] Negotiation successful
0928 03:04:40.502391 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":330,"threads_started":1}
Sep 28 03:04:40 dist-test-slave-fjww krb5kdc[10475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:40.524089 10979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:40.507972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50304 (local address 127.6.192.62:41761)
0928 03:04:40.508202 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:40.508207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:40.508394 (+ 187us) server_negotiation.cc:407] Connection header received
0928 03:04:40.508554 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:40.508558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:40.508624 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:40.508707 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:40.510139 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:40.510920 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:40.512396 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:40.512798 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:40.512911 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:40.517523 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:40.517529 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:40.517533 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:40.517580 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:40.520839 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:40.521782 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:40.521787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:40.521790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:40.521938 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:40.522517 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:40.522521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:40.522523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:40.522914 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:40.523126 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:40.523602 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 03:04:40.523832 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 03:04:40.524786 10520 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50304
I20260928 03:04:40.526093 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 10581
I20260928 03:04:40.540863 10985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:40.527325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57988 (local address 127.6.192.1:37877)
0928 03:04:40.527662 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:40.527667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:40.527742 (+ 75us) server_negotiation.cc:407] Connection header received
0928 03:04:40.527911 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:40.527916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:40.527986 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:40.528080 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:40.529402 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:40.530169 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:40.531434 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:40.532063 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:40.532179 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:40.534595 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:40.534601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:40.534605 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:40.534643 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:40.537970 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:40.538716 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:40.538724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:40.538728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:40.538886 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:40.539285 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:40.539290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:40.539293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:40.539694 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:40.539990 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:40.540270 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 03:04:40.540503 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260928 03:04:40.541532 10704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:40.631094 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 10581
I20260928 03:04:40.649960 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 10715
Sep 28 03:04:40 dist-test-slave-fjww krb5kdc[10475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.2@KRBTEST.COM
I20260928 03:04:40.668362 10990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:40.650993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43366 (local address 127.6.192.2:40873)
0928 03:04:40.651709 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:40.651723 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:40.651771 (+ 48us) server_negotiation.cc:407] Connection header received
0928 03:04:40.651846 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:40.651853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:40.652093 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:40.652258 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:40.653626 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:40.654639 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:40.656357 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:40.656959 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:40.657069 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:40.661848 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:40.661862 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:40.661879 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:40.661920 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:40.665163 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:40.666000 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:40.666009 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:40.666011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:40.666150 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:40.666616 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:40.666620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:40.666622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:40.666971 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:40.667288 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:40.667739 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 03:04:40.667954 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":399,"threads_started":1}
I20260928 03:04:40.670156 10838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:40.758955 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 10715
I20260928 03:04:40.775296 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 10849
Sep 28 03:04:40 dist-test-slave-fjww krb5kdc[10475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.3@KRBTEST.COM
I20260928 03:04:40.794787 10992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:40.776462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53968 (local address 127.6.192.3:44371)
0928 03:04:40.777185 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:40.777193 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:40.777225 (+ 32us) server_negotiation.cc:407] Connection header received
0928 03:04:40.777283 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:40.777291 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:40.777507 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:40.777645 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:40.779076 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:40.780268 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:40.781943 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:40.782624 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:40.782755 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:40.788026 (+ 5271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:40.788043 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:40.788059 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:40.788103 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:40.791123 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:40.792185 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:40.792194 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:40.792197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:40.792436 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:40.792895 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:40.792899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:40.792901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:40.793352 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:40.793628 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:40.794005 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 03:04:40.794442 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":386,"threads_started":1}
I20260928 03:04:40.796186 10972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:40.890388 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 10849
I20260928 03:04:40.906235 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 10488
I20260928 03:04:40.907604 10550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:41.013378 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 10488
2026-09-28T03:04:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4146 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:04:41 dist-test-slave-fjww krb5kdc[10997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:04:41 dist-test-slave-fjww krb5kdc[10997](info): set up 2 sockets
Sep 28 03:04:41 dist-test-slave-fjww krb5kdc[10997](info): commencing operation
krb5kdc: starting...
W20260928 03:04:42.124644 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.005s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:04:42 dist-test-slave-fjww krb5kdc[10997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:04:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:04:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:04:42.347672 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:40119
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:38629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:40119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:42.683353 11010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:42.683769 11010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:42.683871 11010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:42.691416 11010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:04:42.691640 11010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:42.691689 11010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:42.691745 11010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:04:42.691794 11010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:04:42.702370 11010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:40119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:40119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:42.703760 11010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:42.705093 11010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:42.717904 11018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:42.717938 11016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:42.718019 11015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:42.721459 11010 server_base.cc:1034] running on GCE node
I20260928 03:04:42.722493 11010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:42.724267 11010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:42.725536 11010 hybrid_clock.cc:648] HybridClock initialized: now 1790564682725472 us; error 59 us; skew 500 ppm
I20260928 03:04:42.725960 11010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:42 dist-test-slave-fjww krb5kdc[10997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564682, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:42.734143 11010 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:04:42.735661 11010 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:42.735813 11010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:42.736152 11010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:42.739348 11010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "d1385898f2c548178f605457712e73c8"
format_stamp: "Formatted at 2026-09-28 03:04:42 on dist-test-slave-fjww"
server_key: "5044707c82456c0f042f214bc9dad5bd"
server_key_iv: "53f8db251cd25b4024ffa03b3aeb5ab1"
server_key_version: "encryptionkey@0"
I20260928 03:04:42.740301 11010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "d1385898f2c548178f605457712e73c8"
format_stamp: "Formatted at 2026-09-28 03:04:42 on dist-test-slave-fjww"
server_key: "5044707c82456c0f042f214bc9dad5bd"
server_key_iv: "53f8db251cd25b4024ffa03b3aeb5ab1"
server_key_version: "encryptionkey@0"
I20260928 03:04:42.745896 11010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:04:42.749902 11024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:42.751597 11010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:04:42.751822 11010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "d1385898f2c548178f605457712e73c8"
format_stamp: "Formatted at 2026-09-28 03:04:42 on dist-test-slave-fjww"
server_key: "5044707c82456c0f042f214bc9dad5bd"
server_key_iv: "53f8db251cd25b4024ffa03b3aeb5ab1"
server_key_version: "encryptionkey@0"
I20260928 03:04:42.752099 11010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:42.784512 11010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:42.789700 11010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:42.790043 11010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:42.811650 11010 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:40119
I20260928 03:04:42.811690 11075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:40119 every 8 connection(s)
I20260928 03:04:42.812666 11010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:04:42.816805 11076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:42.819273 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 11010
I20260928 03:04:42.819609 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:04:42.820410 6912 external_mini_cluster.cc:1428] Setting key 7a6e5a56a86f46252e050b61e3f0ff97
I20260928 03:04:42.830948 11076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8: Bootstrap starting.
I20260928 03:04:42.835347 11076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:42.836987 11076 log.cc:824] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8: Log is configured to *not* fsync() on all Append() calls
Sep 28 03:04:42 dist-test-slave-fjww krb5kdc[10997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:42.840838 11076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8: No bootstrap required, opened a new log
I20260928 03:04:42.849550 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:42.823185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45136 (local address 127.6.192.62:40119)
0928 03:04:42.824282 (+ 1097us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:42.824297 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:42.824348 (+ 51us) server_negotiation.cc:407] Connection header received
0928 03:04:42.826407 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:42.826451 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:42.826972 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:42.827406 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:42.829521 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:42.831245 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:42.833314 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:42.834088 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:42.834321 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:42.839909 (+ 5588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:42.839923 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:42.839946 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:42.839994 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:42.844330 (+ 4336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:42.845254 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:42.845264 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:42.845273 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:42.845541 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:42.845989 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:42.845995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:42.846000 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:42.846812 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:42.847147 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:42.847562 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 03:04:42.848073 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":571,"threads_started":1}
I20260928 03:04:42.851548 11076 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1385898f2c548178f605457712e73c8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40119 } }
I20260928 03:04:42.852017 11076 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:42.852144 11076 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1385898f2c548178f605457712e73c8, State: Initialized, Role: FOLLOWER
I20260928 03:04:42.852756 11076 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1385898f2c548178f605457712e73c8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40119 } }
I20260928 03:04:42.853029 11076 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:04:42.853202 11076 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:04:42.853389 11076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:42.855679 11076 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1385898f2c548178f605457712e73c8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40119 } }
I20260928 03:04:42.856321 11076 leader_election.cc:302] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1385898f2c548178f605457712e73c8; no voters:
I20260928 03:04:42.856973 11076 leader_election.cc:288] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:04:42.857237 11081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:42.858091 11081 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8 [term 1 LEADER]: Becoming Leader. State: Replica: d1385898f2c548178f605457712e73c8, State: Running, Role: LEADER
I20260928 03:04:42.858734 11081 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1385898f2c548178f605457712e73c8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40119 } }
I20260928 03:04:42.859599 11076 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:04:42.861886 11083 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1385898f2c548178f605457712e73c8. Latest consensus state: current_term: 1 leader_uuid: "d1385898f2c548178f605457712e73c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1385898f2c548178f605457712e73c8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40119 } } }
I20260928 03:04:42.862084 11082 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1385898f2c548178f605457712e73c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1385898f2c548178f605457712e73c8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40119 } } }
I20260928 03:04:42.862398 11083 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:42.862459 11082 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:42.863838 11090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:04:42.870450 11090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:04:42.879912 11090 catalog_manager.cc:1347] Generated new cluster ID: 1dce87dd127a4e2c93c3faa6e0d11633
I20260928 03:04:42.880050 11090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:04:42.890344 11090 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:04:42.892330 11090 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:04:42.900401 11090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8: Generated new TSK 0
I20260928 03:04:42.901403 11090 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
I20260928 03:04:42.968243 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:40119
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:38629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:43.350481 11102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:43.350795 11102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:43.350996 11102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:43.359195 11102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:43.359376 11102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:43.359611 11102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:04:43.370447 11102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:40119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:43.372014 11102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:43.373350 11102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:43.387481 11110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:43.387578 11108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:43.388347 11107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:43.389004 11102 server_base.cc:1034] running on GCE node
I20260928 03:04:43.389703 11102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:43.390805 11102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:43.392063 11102 hybrid_clock.cc:648] HybridClock initialized: now 1790564683391953 us; error 111 us; skew 500 ppm
I20260928 03:04:43.392473 11102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:43 dist-test-slave-fjww krb5kdc[10997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564683, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:43.396015 11102 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:04:43.397302 11102 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:43.397442 11102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:43.397770 11102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:43.401204 11102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "9fdc182a0b1745e5a25ac7c57cf94b32"
format_stamp: "Formatted at 2026-09-28 03:04:43 on dist-test-slave-fjww"
server_key: "ae3e3d4dd2c5b3ab858eb02467abc9d2"
server_key_iv: "46b507f0a51390bd610ce77ee06dc3d6"
server_key_version: "encryptionkey@0"
I20260928 03:04:43.402068 11102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "9fdc182a0b1745e5a25ac7c57cf94b32"
format_stamp: "Formatted at 2026-09-28 03:04:43 on dist-test-slave-fjww"
server_key: "ae3e3d4dd2c5b3ab858eb02467abc9d2"
server_key_iv: "46b507f0a51390bd610ce77ee06dc3d6"
server_key_version: "encryptionkey@0"
I20260928 03:04:43.407569 11102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 03:04:43.411662 11116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:43.413235 11102 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:04:43.413458 11102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "9fdc182a0b1745e5a25ac7c57cf94b32"
format_stamp: "Formatted at 2026-09-28 03:04:43 on dist-test-slave-fjww"
server_key: "ae3e3d4dd2c5b3ab858eb02467abc9d2"
server_key_iv: "46b507f0a51390bd610ce77ee06dc3d6"
server_key_version: "encryptionkey@0"
I20260928 03:04:43.413702 11102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:43.434619 11102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:43.439297 11102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:43.439646 11102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:43.440732 11102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:43.442170 11102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:43.442260 11102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:43.442334 11102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:43.442381 11102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:43.477942 11102 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:41991
I20260928 03:04:43.478031 11228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:41991 every 8 connection(s)
I20260928 03:04:43.478837 11102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:04:43.480906 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 11102
I20260928 03:04:43.481132 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:04:43.481976 6912 external_mini_cluster.cc:1428] Setting key 84141767f8ef9981afa49a0e4d81e3f8
Sep 28 03:04:43 dist-test-slave-fjww krb5kdc[10997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564683, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.192.2@KRBTEST.COM; defaulting to no policy
I20260928 03:04:43.501863 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:43.481870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:43903 (local address 127.6.192.62:40119)
0928 03:04:43.482557 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:43.482562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:43.483608 (+ 1046us) server_negotiation.cc:407] Connection header received
0928 03:04:43.485863 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:43.485870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:43.485973 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:43.486105 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:43.487932 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:43.488690 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:43.490463 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:43.490914 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:43.491066 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:43.495160 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:43.495167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:43.495171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:43.495213 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:43.498199 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:43.499076 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:43.499090 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:43.499093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:43.499229 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:43.499799 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:43.499804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:43.499807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:43.500167 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:43.500332 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:43.501153 (+ 821us) server_negotiation.cc:299] Negotiation successful
0928 03:04:43.501411 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":431,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.2@KRBTEST.COM" created.
I20260928 03:04:43.503041 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:43.482448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:40119 (local address 127.6.192.1:43903)
0928 03:04:43.483362 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:43.483432 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:43.485534 (+ 2102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:43.486377 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:43.486387 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:43.486982 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:43.487726 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:43.487751 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:43.488883 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:43.488889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:43.490270 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:43.490283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:43.491200 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:43.491208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:43.491528 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:43.492345 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:43.492371 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:43.494861 (+ 2490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:43.498384 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:43.498396 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:43.498416 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:43.498918 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:43.499358 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:43.499364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:43.499368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:43.499659 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:43.500328 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:43.500339 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:43.500721 (+ 382us) client_negotiation.cc:769] Sending connection context
0928 03:04:43.501027 (+ 306us) client_negotiation.cc:240] Negotiation successful
0928 03:04:43.501621 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":363,"threads_started":1}
I20260928 03:04:43.504889 11229 heartbeater.cc:347] Connected to a master server at 127.6.192.62:40119
I20260928 03:04:43.505272 11229 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:43.506021 11229 heartbeater.cc:511] Master 127.6.192.62:40119 requested a full tablet report, sending...
I20260928 03:04:43.508388 11041 ts_manager.cc:194] Registered new tserver with Master: 9fdc182a0b1745e5a25ac7c57cf94b32 (127.6.192.1:41991)
I20260928 03:04:43.510085 11041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:43903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
I20260928 03:04:43.532562 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:40119
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:38629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:43.866326 11235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:43.866633 11235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:43.866732 11235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:43.874289 11235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:43.874421 11235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:43.874542 11235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:04:43.884572 11235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:40119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:43.886428 11235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:43.887863 11235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:43.901499 11241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:43.901592 11240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:43.901563 11243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:43.903112 11235 server_base.cc:1034] running on GCE node
I20260928 03:04:43.903927 11235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:43.904906 11235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:43.906114 11235 hybrid_clock.cc:648] HybridClock initialized: now 1790564683906079 us; error 70 us; skew 500 ppm
I20260928 03:04:43.906459 11235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:43 dist-test-slave-fjww krb5kdc[10997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564683, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:43.909934 11235 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:04:43.911301 11235 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:43.911501 11235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:43.911849 11235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:43.915213 11235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "a5b257cadb2a4ac688d429570c95e994"
format_stamp: "Formatted at 2026-09-28 03:04:43 on dist-test-slave-fjww"
server_key: "885696437c64cbb0bc393a74be08ff83"
server_key_iv: "489ef14d43ffd1a79c25b2be8d7d4551"
server_key_version: "encryptionkey@0"
I20260928 03:04:43.916162 11235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "a5b257cadb2a4ac688d429570c95e994"
format_stamp: "Formatted at 2026-09-28 03:04:43 on dist-test-slave-fjww"
server_key: "885696437c64cbb0bc393a74be08ff83"
server_key_iv: "489ef14d43ffd1a79c25b2be8d7d4551"
server_key_version: "encryptionkey@0"
I20260928 03:04:43.921998 11235 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:04:43.926013 11249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:43.927925 11235 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 03:04:43.928190 11235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "a5b257cadb2a4ac688d429570c95e994"
format_stamp: "Formatted at 2026-09-28 03:04:43 on dist-test-slave-fjww"
server_key: "885696437c64cbb0bc393a74be08ff83"
server_key_iv: "489ef14d43ffd1a79c25b2be8d7d4551"
server_key_version: "encryptionkey@0"
I20260928 03:04:43.928439 11235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:43.945060 11235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:43.949792 11235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:43.950086 11235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:43.951099 11235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:43.952677 11235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:43.952769 11235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:43.952930 11235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:43.953011 11235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:43.990203 11235 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:37207
I20260928 03:04:43.990226 11361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:37207 every 8 connection(s)
I20260928 03:04:43.991122 11235 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:04:43.994093 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 11235
I20260928 03:04:43.994294 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:04:43.994765 6912 external_mini_cluster.cc:1428] Setting key a27cbc69564ee19a9613105e9422d5a9
Sep 28 03:04:44 dist-test-slave-fjww krb5kdc[10997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564683, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.192.3@KRBTEST.COM; defaulting to no policy
I20260928 03:04:44.014850 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:43.993834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:48823 (local address 127.6.192.62:40119)
0928 03:04:43.994074 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:43.994081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:43.995365 (+ 1284us) server_negotiation.cc:407] Connection header received
0928 03:04:43.997288 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:43.997295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:43.997398 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:43.997507 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:43.999214 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.000398 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.002050 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.002589 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.002736 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:44.006777 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:44.006786 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:44.006794 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:44.006844 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:44.010779 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:44.011894 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:44.011903 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:44.011909 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:44.012090 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:44.012674 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:44.012680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:44.012683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:44.013076 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:44.013269 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:44.014189 (+ 920us) server_negotiation.cc:299] Negotiation successful
0928 03:04:44.014492 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 03:04:44.016036 11364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:43.994309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:40119 (local address 127.6.192.2:48823)
0928 03:04:43.995195 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:43.995252 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:43.997010 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:43.997806 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:43.997815 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:43.998412 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:43.999013 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:43.999037 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.000564 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.000568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:44.001890 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:44.001902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.002850 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.002858 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:44.003082 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:44.003907 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:44.003947 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:44.006493 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:44.010990 (+ 4497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:44.011000 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:44.011024 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:44.011751 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:44.012311 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:44.012315 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:44.012318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:44.012533 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:44.013336 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:44.013343 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:44.013742 (+ 399us) client_negotiation.cc:769] Sending connection context
0928 03:04:44.014017 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 03:04:44.014577 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":387,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.3@KRBTEST.COM" created.
I20260928 03:04:44.018229 11362 heartbeater.cc:347] Connected to a master server at 127.6.192.62:40119
I20260928 03:04:44.018656 11362 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:44.019411 11362 heartbeater.cc:511] Master 127.6.192.62:40119 requested a full tablet report, sending...
I20260928 03:04:44.021243 11041 ts_manager.cc:194] Registered new tserver with Master: a5b257cadb2a4ac688d429570c95e994 (127.6.192.2:37207)
I20260928 03:04:44.022639 11041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:48823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
I20260928 03:04:44.047657 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:40119
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:38629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:44.386221 11367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:44.386590 11367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:44.386754 11367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:44.394491 11367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:44.394676 11367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:44.394856 11367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:04:44.404906 11367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:40119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:44.406322 11367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:44.407877 11367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:44.421227 11372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:44.421219 11373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:44.421506 11375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:44.422073 11367 server_base.cc:1034] running on GCE node
I20260928 03:04:44.422772 11367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:44.423996 11367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:44.425239 11367 hybrid_clock.cc:648] HybridClock initialized: now 1790564684425170 us; error 65 us; skew 500 ppm
I20260928 03:04:44.425622 11367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:44 dist-test-slave-fjww krb5kdc[10997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564684, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:44.429216 11367 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:04:44.430576 11367 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:44.430734 11367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:44.431092 11367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:44.434470 11367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "540857fff5754192ae5cb7ac4efae042"
format_stamp: "Formatted at 2026-09-28 03:04:44 on dist-test-slave-fjww"
server_key: "d9166b3408d393780df849405f295190"
server_key_iv: "0c6c5810c43147007e5faf90a7654e0a"
server_key_version: "encryptionkey@0"
I20260928 03:04:44.435371 11367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "540857fff5754192ae5cb7ac4efae042"
format_stamp: "Formatted at 2026-09-28 03:04:44 on dist-test-slave-fjww"
server_key: "d9166b3408d393780df849405f295190"
server_key_iv: "0c6c5810c43147007e5faf90a7654e0a"
server_key_version: "encryptionkey@0"
I20260928 03:04:44.441082 11367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 03:04:44.444967 11381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:44.446794 11367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:04:44.447058 11367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "540857fff5754192ae5cb7ac4efae042"
format_stamp: "Formatted at 2026-09-28 03:04:44 on dist-test-slave-fjww"
server_key: "d9166b3408d393780df849405f295190"
server_key_iv: "0c6c5810c43147007e5faf90a7654e0a"
server_key_version: "encryptionkey@0"
I20260928 03:04:44.447348 11367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:44.473793 11367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:44.479213 11367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:44.479703 11367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:44.480930 11367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:44.482532 11367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:44.482671 11367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:44.482832 11367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:44.482933 11367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:44.513805 11229 heartbeater.cc:503] Master 127.6.192.62:40119 was elected leader, sending a full tablet report...
I20260928 03:04:44.522693 11367 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:37309
I20260928 03:04:44.522797 11493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:37309 every 8 connection(s)
I20260928 03:04:44.523754 11367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:04:44.530308 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 11367
I20260928 03:04:44.530539 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:04:44.531055 6912 external_mini_cluster.cc:1428] Setting key f33c411e22f9b95227d2636a75037bba
Sep 28 03:04:44 dist-test-slave-fjww krb5kdc[10997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564684, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:44.546561 11496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:44.526640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:49503 (local address 127.6.192.62:40119)
0928 03:04:44.527149 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:44.527156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:44.528324 (+ 1168us) server_negotiation.cc:407] Connection header received
0928 03:04:44.530313 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:44.530321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:44.530442 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:44.530582 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:44.532749 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.533731 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.535695 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.536152 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.536296 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:44.540182 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:44.540187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:44.540191 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:44.540227 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:44.542757 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:44.543711 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:44.543715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:44.543717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:44.543853 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:44.544310 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:44.544313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:44.544315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:44.544615 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:44.544727 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:44.545856 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0928 03:04:44.546140 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":227,"threads_started":1}
I20260928 03:04:44.547853 11497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:44.527301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:40119 (local address 127.6.192.3:49503)
0928 03:04:44.528130 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:44.528175 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:44.529886 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:44.530850 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:44.530863 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:44.531656 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:44.532531 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:44.532561 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.533903 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.533909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:44.535505 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:44.535523 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.536399 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.536407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:44.536685 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:44.537412 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:44.537436 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:44.539931 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:44.542913 (+ 2982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:44.542921 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:44.542936 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:44.543585 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:44.543997 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:44.544001 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:44.544003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:44.544188 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:44.544782 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:44.544790 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:44.545344 (+ 554us) client_negotiation.cc:769] Sending connection context
0928 03:04:44.545739 (+ 395us) client_negotiation.cc:240] Negotiation successful
0928 03:04:44.546386 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":359,"threads_started":1}
I20260928 03:04:44.549350 11494 heartbeater.cc:347] Connected to a master server at 127.6.192.62:40119
I20260928 03:04:44.549688 11494 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:44.550390 11494 heartbeater.cc:511] Master 127.6.192.62:40119 requested a full tablet report, sending...
I20260928 03:04:44.552004 11041 ts_manager.cc:194] Registered new tserver with Master: 540857fff5754192ae5cb7ac4efae042 (127.6.192.3:37309)
I20260928 03:04:44.552922 11041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:49503
I20260928 03:04:44.560197 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:04:44.578790 11496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:44.565146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45144 (local address 127.6.192.62:40119)
0928 03:04:44.565394 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:44.565404 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:44.565893 (+ 489us) server_negotiation.cc:407] Connection header received
0928 03:04:44.566087 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:44.566093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:44.566177 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:44.566275 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:44.567889 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.568712 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.570077 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.570571 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.570685 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:44.573093 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:44.573098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:44.573102 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:44.573142 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:44.575871 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:44.576544 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:44.576548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:44.576551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:44.576685 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:44.577055 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:44.577059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:44.577061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:44.577424 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:44.577690 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:44.578155 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 03:04:44.578372 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 03:04:44.583415 11041 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:04:44.591218 11041 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:04:44.636122 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:44.607103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51096 (local address 127.6.192.1:41991)
0928 03:04:44.626263 (+ 19160us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:44.626277 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:44.626330 (+ 53us) server_negotiation.cc:407] Connection header received
0928 03:04:44.626482 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:44.626495 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:44.626919 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:44.627207 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:44.628091 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.629737 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.632800 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.634449 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.634593 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:44.634888 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:44.635577 (+ 689us) server_negotiation.cc:299] Negotiation successful
0928 03:04:44.635768 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18913,"thread_start_us":18708,"threads_started":1}
I20260928 03:04:44.636695 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:44.605969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41991 (local address 127.0.0.1:51096)
0928 03:04:44.623164 (+ 17195us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:44.623222 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:44.623480 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:44.627241 (+ 3761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:44.627249 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:44.627299 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:44.627821 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:44.627839 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.629921 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.629929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:44.632601 (+ 2672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:44.632616 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.634666 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.634672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:44.634908 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:44.634962 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 03:04:44.635110 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 03:04:44.635326 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16916,"thread_start_us":16687,"threads_started":1}
I20260928 03:04:44.637673 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:44.605973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:37309 (local address 127.0.0.1:55974)
0928 03:04:44.628010 (+ 22037us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:44.628032 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:44.628143 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:44.629654 (+ 1511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:44.629658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:44.629684 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:44.630401 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:44.630418 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.632631 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.632636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:44.634473 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:44.634487 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.636485 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.636491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:44.636695 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:44.636721 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 03:04:44.636805 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 03:04:44.637279 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21876,"mutex_wait_us":78,"thread_start_us":22330,"threads_started":1}
I20260928 03:04:44.637921 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:44.606624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.6.192.2:37207)
0928 03:04:44.625359 (+ 18735us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:44.625374 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:44.628778 (+ 3404us) server_negotiation.cc:407] Connection header received
0928 03:04:44.629074 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:44.629082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:44.629359 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:44.629527 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:44.630786 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.632442 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.635358 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.636815 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.636947 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:44.637153 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:44.637420 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 03:04:44.637577 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18462,"thread_start_us":18755,"threads_started":1}
I20260928 03:04:44.638396 11508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:44.606289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:37207 (local address 127.0.0.1:58670)
0928 03:04:44.628651 (+ 22362us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:44.628670 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:44.628888 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:44.629830 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:44.629837 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:44.629890 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:44.630591 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:44.630605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.632710 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.632717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:44.635140 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:44.635155 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.637038 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.637044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:44.637213 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:44.637245 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 03:04:44.637366 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 03:04:44.637502 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22232,"mutex_wait_us":1,"spinlock_wait_cycles":1408,"thread_start_us":23660,"threads_started":1}
I20260928 03:04:44.638727 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:44.606277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55974 (local address 127.6.192.3:37309)
0928 03:04:44.628861 (+ 22584us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:44.628870 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:44.628903 (+ 33us) server_negotiation.cc:407] Connection header received
0928 03:04:44.629265 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:44.629270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:44.629450 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:44.629661 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:44.630623 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.632429 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.634679 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.636232 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.637810 (+ 1578us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:44.638060 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:44.638131 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 03:04:44.638346 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22380,"thread_start_us":22738,"threads_started":1}
I20260928 03:04:44.641189 11164 tablet_service.cc:1467] Processing CreateTablet for tablet 4002f3d8103c4a48991a8bf0d4fe5a8d (DEFAULT_TABLE table=test-table [id=c3edc000ad19434a8920b876c3008fea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:44.641952 11429 tablet_service.cc:1467] Processing CreateTablet for tablet 4002f3d8103c4a48991a8bf0d4fe5a8d (DEFAULT_TABLE table=test-table [id=c3edc000ad19434a8920b876c3008fea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:44.642035 11297 tablet_service.cc:1467] Processing CreateTablet for tablet 4002f3d8103c4a48991a8bf0d4fe5a8d (DEFAULT_TABLE table=test-table [id=c3edc000ad19434a8920b876c3008fea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:44.642768 11164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4002f3d8103c4a48991a8bf0d4fe5a8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:44.643208 11429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4002f3d8103c4a48991a8bf0d4fe5a8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:44.643354 11297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4002f3d8103c4a48991a8bf0d4fe5a8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:44.656216 11513 tablet_bootstrap.cc:492] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042: Bootstrap starting.
I20260928 03:04:44.661459 11513 tablet_bootstrap.cc:654] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:44.662786 11513 log.cc:824] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:44.665575 11515 tablet_bootstrap.cc:492] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32: Bootstrap starting.
I20260928 03:04:44.666397 11513 tablet_bootstrap.cc:492] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042: No bootstrap required, opened a new log
I20260928 03:04:44.666702 11513 ts_tablet_manager.cc:1397] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260928 03:04:44.667577 11514 tablet_bootstrap.cc:492] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994: Bootstrap starting.
I20260928 03:04:44.669775 11515 tablet_bootstrap.cc:654] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:44.671226 11515 log.cc:824] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:44.671872 11514 tablet_bootstrap.cc:654] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:44.673111 11514 log.cc:824] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:44.674826 11515 tablet_bootstrap.cc:492] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32: No bootstrap required, opened a new log
I20260928 03:04:44.675132 11515 ts_tablet_manager.cc:1397] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 03:04:44.675710 11513 raft_consensus.cc:348] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540857fff5754192ae5cb7ac4efae042" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 37309 } } peers { permanent_uuid: "9fdc182a0b1745e5a25ac7c57cf94b32" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41991 } } peers { permanent_uuid: "a5b257cadb2a4ac688d429570c95e994" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37207 } }
I20260928 03:04:44.676107 11513 raft_consensus.cc:374] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:44.676257 11513 raft_consensus.cc:729] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540857fff5754192ae5cb7ac4efae042, State: Initialized, Role: FOLLOWER
I20260928 03:04:44.676375 11514 tablet_bootstrap.cc:492] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994: No bootstrap required, opened a new log
I20260928 03:04:44.676658 11514 ts_tablet_manager.cc:1397] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260928 03:04:44.676931 11513 consensus_queue.cc:260] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540857fff5754192ae5cb7ac4efae042" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 37309 } } peers { permanent_uuid: "9fdc182a0b1745e5a25ac7c57cf94b32" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41991 } } peers { permanent_uuid: "a5b257cadb2a4ac688d429570c95e994" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37207 } }
I20260928 03:04:44.678135 11494 heartbeater.cc:503] Master 127.6.192.62:40119 was elected leader, sending a full tablet report...
I20260928 03:04:44.678982 11513 ts_tablet_manager.cc:1428] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260928 03:04:44.686302 11515 raft_consensus.cc:348] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540857fff5754192ae5cb7ac4efae042" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 37309 } } peers { permanent_uuid: "9fdc182a0b1745e5a25ac7c57cf94b32" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41991 } } peers { permanent_uuid: "a5b257cadb2a4ac688d429570c95e994" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37207 } }
I20260928 03:04:44.686841 11515 raft_consensus.cc:374] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:44.686954 11515 raft_consensus.cc:729] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fdc182a0b1745e5a25ac7c57cf94b32, State: Initialized, Role: FOLLOWER
I20260928 03:04:44.687657 11515 consensus_queue.cc:260] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540857fff5754192ae5cb7ac4efae042" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 37309 } } peers { permanent_uuid: "9fdc182a0b1745e5a25ac7c57cf94b32" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41991 } } peers { permanent_uuid: "a5b257cadb2a4ac688d429570c95e994" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37207 } }
I20260928 03:04:44.688041 11514 raft_consensus.cc:348] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540857fff5754192ae5cb7ac4efae042" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 37309 } } peers { permanent_uuid: "9fdc182a0b1745e5a25ac7c57cf94b32" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41991 } } peers { permanent_uuid: "a5b257cadb2a4ac688d429570c95e994" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37207 } }
I20260928 03:04:44.688522 11514 raft_consensus.cc:374] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:44.688623 11514 raft_consensus.cc:729] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b257cadb2a4ac688d429570c95e994, State: Initialized, Role: FOLLOWER
I20260928 03:04:44.689291 11514 consensus_queue.cc:260] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540857fff5754192ae5cb7ac4efae042" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 37309 } } peers { permanent_uuid: "9fdc182a0b1745e5a25ac7c57cf94b32" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41991 } } peers { permanent_uuid: "a5b257cadb2a4ac688d429570c95e994" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37207 } }
I20260928 03:04:44.689610 11515 ts_tablet_manager.cc:1428] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260928 03:04:44.690452 11362 heartbeater.cc:503] Master 127.6.192.62:40119 was elected leader, sending a full tablet report...
I20260928 03:04:44.690953 11514 ts_tablet_manager.cc:1428] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260928 03:04:44.718694 11521 raft_consensus.cc:482] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:04:44.719031 11521 raft_consensus.cc:504] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540857fff5754192ae5cb7ac4efae042" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 37309 } } peers { permanent_uuid: "9fdc182a0b1745e5a25ac7c57cf94b32" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41991 } } peers { permanent_uuid: "a5b257cadb2a4ac688d429570c95e994" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37207 } }
I20260928 03:04:44.720949 11521 leader_election.cc:288] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 540857fff5754192ae5cb7ac4efae042 (127.6.192.3:37309), 9fdc182a0b1745e5a25ac7c57cf94b32 (127.6.192.1:41991)
I20260928 03:04:44.728051 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:44.721615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:48807 (local address 127.6.192.1:41991)
0928 03:04:44.721809 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:44.721817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:44.722439 (+ 622us) server_negotiation.cc:407] Connection header received
0928 03:04:44.722686 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:44.722690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:44.722751 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:44.722869 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:44.723491 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.724267 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.726172 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.726957 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.727125 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:44.727240 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:44.727632 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 03:04:44.727762 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 03:04:44.728228 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:44.721483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:56165 (local address 127.6.192.3:37309)
0928 03:04:44.721745 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:44.721754 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:44.722587 (+ 833us) server_negotiation.cc:407] Connection header received
0928 03:04:44.722804 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:44.722813 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:44.722876 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:44.722973 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:04:44.723744 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.724492 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.726328 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.727145 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.727303 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:44.727480 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:44.727810 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 03:04:44.727953 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 03:04:44.728282 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:44.721292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41991 (local address 127.6.192.2:48807)
0928 03:04:44.722300 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:44.722320 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:44.722523 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:44.722887 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:44.722891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:44.722917 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:44.723313 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:44.723323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.724409 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.724418 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:44.726001 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:44.726015 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.727194 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.727200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:44.727381 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:44.727430 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 03:04:44.727599 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 03:04:44.727752 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":5,"thread_start_us":515,"threads_started":1}
I20260928 03:04:44.728410 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:44.721302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:37309 (local address 127.6.192.2:56165)
0928 03:04:44.722452 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:44.722468 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:44.722653 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:44.723001 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:44.723006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:44.723050 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:04:44.723548 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:44.723558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.724643 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.724649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:44.726142 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:44.726154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.727370 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.727377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:44.727649 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:44.727673 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 03:04:44.727784 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 03:04:44.727922 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"mutex_wait_us":130,"thread_start_us":618,"threads_started":1}
I20260928 03:04:44.729528 11184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4002f3d8103c4a48991a8bf0d4fe5a8d" candidate_uuid: "a5b257cadb2a4ac688d429570c95e994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fdc182a0b1745e5a25ac7c57cf94b32" is_pre_election: true
I20260928 03:04:44.729673 11449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4002f3d8103c4a48991a8bf0d4fe5a8d" candidate_uuid: "a5b257cadb2a4ac688d429570c95e994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540857fff5754192ae5cb7ac4efae042" is_pre_election: true
I20260928 03:04:44.730202 11449 raft_consensus.cc:2454] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b257cadb2a4ac688d429570c95e994 in term 0.
I20260928 03:04:44.730176 11184 raft_consensus.cc:2454] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b257cadb2a4ac688d429570c95e994 in term 0.
I20260928 03:04:44.730942 11251 leader_election.cc:302] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 540857fff5754192ae5cb7ac4efae042, 9fdc182a0b1745e5a25ac7c57cf94b32, a5b257cadb2a4ac688d429570c95e994; no voters:
W20260928 03:04:44.731324 11230 tablet.cc:2369] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:04:44.731501 11521 raft_consensus.cc:2781] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:04:44.731703 11521 raft_consensus.cc:482] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:04:44.731830 11521 raft_consensus.cc:3037] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:44.734396 11521 raft_consensus.cc:504] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540857fff5754192ae5cb7ac4efae042" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 37309 } } peers { permanent_uuid: "9fdc182a0b1745e5a25ac7c57cf94b32" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41991 } } peers { permanent_uuid: "a5b257cadb2a4ac688d429570c95e994" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37207 } }
I20260928 03:04:44.735129 11521 leader_election.cc:288] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [CANDIDATE]: Term 1 election: Requested vote from peers 540857fff5754192ae5cb7ac4efae042 (127.6.192.3:37309), 9fdc182a0b1745e5a25ac7c57cf94b32 (127.6.192.1:41991)
I20260928 03:04:44.735795 11184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4002f3d8103c4a48991a8bf0d4fe5a8d" candidate_uuid: "a5b257cadb2a4ac688d429570c95e994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fdc182a0b1745e5a25ac7c57cf94b32"
I20260928 03:04:44.735800 11449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4002f3d8103c4a48991a8bf0d4fe5a8d" candidate_uuid: "a5b257cadb2a4ac688d429570c95e994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540857fff5754192ae5cb7ac4efae042"
I20260928 03:04:44.736042 11184 raft_consensus.cc:3037] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:44.736052 11449 raft_consensus.cc:3037] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:44.739481 11184 raft_consensus.cc:2454] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b257cadb2a4ac688d429570c95e994 in term 1.
I20260928 03:04:44.739478 11449 raft_consensus.cc:2454] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b257cadb2a4ac688d429570c95e994 in term 1.
I20260928 03:04:44.740082 11251 leader_election.cc:302] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fdc182a0b1745e5a25ac7c57cf94b32, a5b257cadb2a4ac688d429570c95e994; no voters:
I20260928 03:04:44.740447 11521 raft_consensus.cc:2781] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:44.741036 11521 raft_consensus.cc:686] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [term 1 LEADER]: Becoming Leader. State: Replica: a5b257cadb2a4ac688d429570c95e994, State: Running, Role: LEADER
I20260928 03:04:44.741565 11521 consensus_queue.cc:237] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540857fff5754192ae5cb7ac4efae042" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 37309 } } peers { permanent_uuid: "9fdc182a0b1745e5a25ac7c57cf94b32" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41991 } } peers { permanent_uuid: "a5b257cadb2a4ac688d429570c95e994" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37207 } }
W20260928 03:04:44.742996 11363 tablet.cc:2369] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:04:44.746372 11041 catalog_manager.cc:5495] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5b257cadb2a4ac688d429570c95e994 (127.6.192.2). New cstate: current_term: 1 leader_uuid: "a5b257cadb2a4ac688d429570c95e994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540857fff5754192ae5cb7ac4efae042" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 37309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fdc182a0b1745e5a25ac7c57cf94b32" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5b257cadb2a4ac688d429570c95e994" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37207 } health_report { overall_health: HEALTHY } } }
I20260928 03:04:44.767987 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:44.761773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.6.192.2:37207)
0928 03:04:44.761993 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:44.762002 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:44.762148 (+ 146us) server_negotiation.cc:407] Connection header received
0928 03:04:44.762367 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:44.762371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:44.762454 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:44.762607 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:04:44.762958 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.764154 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.765700 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.766103 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.766219 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:44.766733 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:04:44.767308 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:04:44.767525 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:44.767582 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 03:04:44.767660 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 03:04:44.775648 11495 tablet.cc:2369] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:04:44.777051 11449 raft_consensus.cc:1261] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042 [term 1 FOLLOWER]: Refusing update from remote peer a5b257cadb2a4ac688d429570c95e994: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:44.777051 11184 raft_consensus.cc:1261] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32 [term 1 FOLLOWER]: Refusing update from remote peer a5b257cadb2a4ac688d429570c95e994: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:44.778291 11524 consensus_queue.cc:1035] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "540857fff5754192ae5cb7ac4efae042" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 37309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:04:44.778679 11521 consensus_queue.cc:1035] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fdc182a0b1745e5a25ac7c57cf94b32" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:04:44.809782 11041 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:04:44.810118 11041 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:04:44.814800 11041 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1385898f2c548178f605457712e73c8: Sending DeleteTablet for 3 replicas of tablet 4002f3d8103c4a48991a8bf0d4fe5a8d
I20260928 03:04:44.816269 11429 tablet_service.cc:1514] Processing DeleteTablet for tablet 4002f3d8103c4a48991a8bf0d4fe5a8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:44 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:55974
I20260928 03:04:44.816321 11164 tablet_service.cc:1514] Processing DeleteTablet for tablet 4002f3d8103c4a48991a8bf0d4fe5a8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:44 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:51096
I20260928 03:04:44.817492 11538 tablet_replica.cc:330] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042: stopping tablet replica
I20260928 03:04:44.818084 11537 tablet_replica.cc:330] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32: stopping tablet replica
I20260928 03:04:44.818668 11537 raft_consensus.cc:2229] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:04:44.818893 11297 tablet_service.cc:1514] Processing DeleteTablet for tablet 4002f3d8103c4a48991a8bf0d4fe5a8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:44 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:58670
I20260928 03:04:44.819258 11538 raft_consensus.cc:2229] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:04:44.819849 11538 raft_consensus.cc:2258] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:44.820405 11537 raft_consensus.cc:2258] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:44.822795 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 11102
I20260928 03:04:44.832471 11539 tablet_replica.cc:330] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994: stopping tablet replica
I20260928 03:04:44.833165 11539 raft_consensus.cc:2229] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:04:44.833897 11538 ts_tablet_manager.cc:1905] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:44.834178 11539 raft_consensus.cc:2258] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:44.834745 11537 ts_tablet_manager.cc:1905] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:44.839857 11539 ts_tablet_manager.cc:1905] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:44.842216 11538 ts_tablet_manager.cc:1918] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:04:44.842335 11537 ts_tablet_manager.cc:1918] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:04:44.842515 11538 log.cc:1196] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/4002f3d8103c4a48991a8bf0d4fe5a8d
I20260928 03:04:44.842855 11537 log.cc:1196] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/4002f3d8103c4a48991a8bf0d4fe5a8d
I20260928 03:04:44.843169 11538 ts_tablet_manager.cc:1939] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 540857fff5754192ae5cb7ac4efae042: Deleting consensus metadata
I20260928 03:04:44.843508 11537 ts_tablet_manager.cc:1939] T 4002f3d8103c4a48991a8bf0d4fe5a8d P 9fdc182a0b1745e5a25ac7c57cf94b32: Deleting consensus metadata
I20260928 03:04:44.845441 11026 catalog_manager.cc:4841] TS 9fdc182a0b1745e5a25ac7c57cf94b32 (127.6.192.1:41991): tablet 4002f3d8103c4a48991a8bf0d4fe5a8d (table test-table [id=c3edc000ad19434a8920b876c3008fea]) successfully deleted
I20260928 03:04:44.845798 11025 catalog_manager.cc:4841] TS 540857fff5754192ae5cb7ac4efae042 (127.6.192.3:37309): tablet 4002f3d8103c4a48991a8bf0d4fe5a8d (table test-table [id=c3edc000ad19434a8920b876c3008fea]) successfully deleted
I20260928 03:04:44.850848 11539 ts_tablet_manager.cc:1918] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:04:44.852428 11539 log.cc:1196] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/4002f3d8103c4a48991a8bf0d4fe5a8d
I20260928 03:04:44.853216 11539 ts_tablet_manager.cc:1939] T 4002f3d8103c4a48991a8bf0d4fe5a8d P a5b257cadb2a4ac688d429570c95e994: Deleting consensus metadata
I20260928 03:04:44.855587 11027 catalog_manager.cc:4841] TS a5b257cadb2a4ac688d429570c95e994 (127.6.192.2:37207): tablet 4002f3d8103c4a48991a8bf0d4fe5a8d (table test-table [id=c3edc000ad19434a8920b876c3008fea]) successfully deleted
Sep 28 03:04:44 dist-test-slave-fjww krb5kdc[10997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:04:44.864652 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:44.824733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51112 (local address 127.6.192.1:41991)
0928 03:04:44.824939 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:44.824946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:44.835139 (+ 10193us) server_negotiation.cc:407] Connection header received
0928 03:04:44.835486 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:44.835494 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:44.835583 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:44.835662 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:44.840146 (+ 4484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.841467 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.846238 (+ 4771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.847000 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.847146 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:44.857665 (+ 10519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:44.857681 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:44.857699 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:44.857747 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:44.861302 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:44.862131 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:44.862140 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:44.862143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:44.862307 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:44.862745 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:44.862751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:44.862762 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:44.863167 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:44.863553 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:44.863997 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 03:04:44.864251 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 03:04:44.865577 11224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:44.959726 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 11102
I20260928 03:04:44.979689 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 11235
Sep 28 03:04:44 dist-test-slave-fjww krb5kdc[10997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.2@KRBTEST.COM
I20260928 03:04:44.997550 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:44.980516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58692 (local address 127.6.192.2:37207)
0928 03:04:44.980706 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:44.980713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:44.980733 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:04:44.981024 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:44.981029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:44.981107 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:44.981216 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:44.982430 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.983261 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.984563 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:44.985012 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:44.985155 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:44.989512 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:44.989531 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:44.989553 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:44.989601 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:44.994076 (+ 4475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:44.995043 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:44.995053 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:44.995057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:44.995240 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:44.995716 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:44.995722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:44.995726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:44.996238 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:44.996600 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:44.996873 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 03:04:44.997120 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 03:04:44.998371 11357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:45.116551 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 11235
I20260928 03:04:45.135284 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 11367
Sep 28 03:04:45 dist-test-slave-fjww krb5kdc[10997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.3@KRBTEST.COM
I20260928 03:04:45.151929 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:45.136219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.6.192.3:37309)
0928 03:04:45.136435 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:45.136444 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:45.136465 (+ 21us) server_negotiation.cc:407] Connection header received
0928 03:04:45.136678 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:45.136684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:45.136796 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:45.136912 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:45.138335 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:45.139095 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:45.140474 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:45.140907 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:45.141051 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:45.145634 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:45.145652 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:45.145667 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:45.145710 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:45.148974 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:45.149780 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:45.149789 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:45.149791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:45.149969 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:45.150335 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:45.150339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:45.150341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:45.150722 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:45.150992 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:45.151261 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 03:04:45.151520 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 03:04:45.152788 11489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:45.246450 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 11367
I20260928 03:04:45.263608 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 11010
I20260928 03:04:45.265488 11071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:45.381537 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 11010
2026-09-28T03:04:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4367 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T03:04:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:04:45Z Disabled control of system clock
I20260928 03:04:45.452564 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:39213
--webserver_interface=127.6.192.62
--webserver_port=0
--builtin_ntp_servers=127.6.192.20:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:39213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 03:04:45.807945 11548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:45.808380 11548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:45.808497 11548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:45.816406 11548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:04:45.816551 11548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:45.816594 11548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:45.816651 11548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:04:45.816694 11548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:04:45.826576 11548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:39213
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:39213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:45.827971 11548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:45.829823 11548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:45.842857 11554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:45.842971 11553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:45.842949 11556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:45.843696 11548 server_base.cc:1034] running on GCE node
I20260928 03:04:45.844511 11548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:45.846033 11548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:45.847285 11548 hybrid_clock.cc:648] HybridClock initialized: now 1790564685847236 us; error 42 us; skew 500 ppm
I20260928 03:04:45.847759 11548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:04:45.850274 11548 webserver.cc:460] Webserver started at http://127.6.192.62:32817/ using document root <none> and password file <none>
I20260928 03:04:45.851104 11548 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:45.851199 11548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:45.851608 11548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:45.854665 11548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "44c0ab683445499d9f14f7f240616d25"
format_stamp: "Formatted at 2026-09-28 03:04:45 on dist-test-slave-fjww"
server_key: "c2a15e047d5676a048b70ef79adccb32"
server_key_iv: "ef42924f0f919ff3521ea700997ef049"
server_key_version: "encryptionkey@0"
I20260928 03:04:45.855585 11548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "44c0ab683445499d9f14f7f240616d25"
format_stamp: "Formatted at 2026-09-28 03:04:45 on dist-test-slave-fjww"
server_key: "c2a15e047d5676a048b70ef79adccb32"
server_key_iv: "ef42924f0f919ff3521ea700997ef049"
server_key_version: "encryptionkey@0"
I20260928 03:04:45.861148 11548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 03:04:45.864857 11562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:45.866416 11548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:04:45.866631 11548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "44c0ab683445499d9f14f7f240616d25"
format_stamp: "Formatted at 2026-09-28 03:04:45 on dist-test-slave-fjww"
server_key: "c2a15e047d5676a048b70ef79adccb32"
server_key_iv: "ef42924f0f919ff3521ea700997ef049"
server_key_version: "encryptionkey@0"
I20260928 03:04:45.866888 11548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:45.873128 11548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:45.874004 11548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:45.874343 11548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:45.893846 11548 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:39213
I20260928 03:04:45.893905 11613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:39213 every 8 connection(s)
I20260928 03:04:45.895323 11548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:04:45.899533 11614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:45.905540 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 11548
I20260928 03:04:45.905740 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:04:45.906610 6912 external_mini_cluster.cc:1428] Setting key e88b742e577c5c8a629d24ddb0f6e118
I20260928 03:04:45.912283 11614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25: Bootstrap starting.
I20260928 03:04:45.915834 11614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:45.916000 11617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:45.909029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.6.192.62:39213)
0928 03:04:45.910173 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:45.910186 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:45.910233 (+ 47us) server_negotiation.cc:407] Connection header received
0928 03:04:45.912085 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:45.912110 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:45.912637 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:45.913048 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:45.914226 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:45.914232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:45.914245 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:04:45.914259 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:45.914331 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:04:45.914349 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:45.914582 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:45.914717 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 03:04:45.914984 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":370,"threads_started":1}
I20260928 03:04:45.917001 11614 log.cc:824] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:45.920184 11614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25: No bootstrap required, opened a new log
I20260928 03:04:45.928272 11614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c0ab683445499d9f14f7f240616d25" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 39213 } }
I20260928 03:04:45.928715 11614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:45.928836 11614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c0ab683445499d9f14f7f240616d25, State: Initialized, Role: FOLLOWER
I20260928 03:04:45.929463 11614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c0ab683445499d9f14f7f240616d25" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 39213 } }
I20260928 03:04:45.929735 11614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:04:45.929898 11614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:04:45.930094 11614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:45.932740 11614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c0ab683445499d9f14f7f240616d25" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 39213 } }
I20260928 03:04:45.933311 11614 leader_election.cc:302] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44c0ab683445499d9f14f7f240616d25; no voters:
I20260928 03:04:45.933977 11614 leader_election.cc:288] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:04:45.934132 11619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:45.934888 11619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25 [term 1 LEADER]: Becoming Leader. State: Replica: 44c0ab683445499d9f14f7f240616d25, State: Running, Role: LEADER
I20260928 03:04:45.935611 11619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c0ab683445499d9f14f7f240616d25" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 39213 } }
I20260928 03:04:45.936331 11614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:04:45.938406 11620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44c0ab683445499d9f14f7f240616d25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c0ab683445499d9f14f7f240616d25" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 39213 } } }
I20260928 03:04:45.938804 11620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:45.938499 11621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44c0ab683445499d9f14f7f240616d25. Latest consensus state: current_term: 1 leader_uuid: "44c0ab683445499d9f14f7f240616d25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c0ab683445499d9f14f7f240616d25" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 39213 } } }
I20260928 03:04:45.940452 11621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:45.940342 11626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:04:45.946525 11626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:04:45.957513 11626 catalog_manager.cc:1347] Generated new cluster ID: a4d2269d3c88448680bdb2a47e52477e
I20260928 03:04:45.957646 11626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:04:46.005455 11626 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:04:46.005965 11626 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:04:46.014770 11626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25: Generated new TSK 0
I20260928 03:04:46.015950 11626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:04:46.036592 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:39213
--builtin_ntp_servers=127.6.192.20:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 03:04:46.380292 11638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:46.380606 11638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:46.380751 11638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:46.388509 11638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:46.388625 11638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:46.388813 11638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:04:46.398171 11638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:39213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:46.399578 11638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:46.401428 11638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:46.415128 11646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:46.415165 11644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:46.415128 11643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:46.416337 11638 server_base.cc:1034] running on GCE node
I20260928 03:04:46.417059 11638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:46.418058 11638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:46.419307 11638 hybrid_clock.cc:648] HybridClock initialized: now 1790564686419216 us; error 75 us; skew 500 ppm
I20260928 03:04:46.419691 11638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:04:46.422151 11638 webserver.cc:460] Webserver started at http://127.6.192.1:34315/ using document root <none> and password file <none>
I20260928 03:04:46.422940 11638 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:46.423096 11638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:46.423422 11638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:46.426442 11638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "4712fa96315c49ae84f3e000484c875b"
format_stamp: "Formatted at 2026-09-28 03:04:46 on dist-test-slave-fjww"
server_key: "dd3c38a6941e86cd2403978715570248"
server_key_iv: "9342c8e1adec823e29bdbe2a64f8fd24"
server_key_version: "encryptionkey@0"
I20260928 03:04:46.427263 11638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "4712fa96315c49ae84f3e000484c875b"
format_stamp: "Formatted at 2026-09-28 03:04:46 on dist-test-slave-fjww"
server_key: "dd3c38a6941e86cd2403978715570248"
server_key_iv: "9342c8e1adec823e29bdbe2a64f8fd24"
server_key_version: "encryptionkey@0"
I20260928 03:04:46.432756 11638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 03:04:46.436507 11652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:46.438330 11638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:04:46.438522 11638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "4712fa96315c49ae84f3e000484c875b"
format_stamp: "Formatted at 2026-09-28 03:04:46 on dist-test-slave-fjww"
server_key: "dd3c38a6941e86cd2403978715570248"
server_key_iv: "9342c8e1adec823e29bdbe2a64f8fd24"
server_key_version: "encryptionkey@0"
I20260928 03:04:46.438802 11638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:46.445222 11638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:46.446120 11638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:46.446416 11638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:46.447373 11638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:46.448999 11638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:46.449120 11638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:46.449256 11638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:46.449344 11638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:46.483520 11638 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:41019
I20260928 03:04:46.483579 11764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:41019 every 8 connection(s)
I20260928 03:04:46.484942 11638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:04:46.488493 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 11638
I20260928 03:04:46.488684 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:04:46.489564 6912 external_mini_cluster.cc:1428] Setting key f716128cbe34ace70e29bdad3f7d2862
I20260928 03:04:46.492727 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:39213
--builtin_ntp_servers=127.6.192.20:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 03:04:46.495345 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:46.488049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:51821 (local address 127.6.192.62:39213)
0928 03:04:46.488508 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:46.488513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:46.490393 (+ 1880us) server_negotiation.cc:407] Connection header received
0928 03:04:46.492504 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:46.492513 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:46.492587 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:46.492719 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:46.494400 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:46.494408 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:46.494416 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:04:46.494441 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:46.494501 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:04:46.494520 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:46.494650 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:46.494877 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 03:04:46.494954 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":268,"threads_started":1}
I20260928 03:04:46.496309 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:46.488836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:39213 (local address 127.6.192.1:51821)
0928 03:04:46.490168 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:46.490247 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:46.492169 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:46.493128 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:46.493138 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:46.493261 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:46.494017 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:04:46.494020 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:46.494064 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:04:46.494178 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:46.494631 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:46.494640 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:46.494646 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 03:04:46.494919 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 03:04:46.495160 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":443,"threads_started":1}
I20260928 03:04:46.498169 11765 heartbeater.cc:347] Connected to a master server at 127.6.192.62:39213
I20260928 03:04:46.498517 11765 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:46.499172 11765 heartbeater.cc:511] Master 127.6.192.62:39213 requested a full tablet report, sending...
I20260928 03:04:46.501482 11579 ts_manager.cc:194] Registered new tserver with Master: 4712fa96315c49ae84f3e000484c875b (127.6.192.1:41019)
W20260928 03:04:46.845011 11769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:46.845355 11769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:46.845533 11769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:46.853380 11769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:46.853545 11769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:46.853736 11769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:04:46.863409 11769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:39213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:46.864912 11769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:46.866786 11769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:46.881608 11774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:46.881536 11777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:46.881505 11775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:46.882267 11769 server_base.cc:1034] running on GCE node
I20260928 03:04:46.883092 11769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:46.884351 11769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:46.885583 11769 hybrid_clock.cc:648] HybridClock initialized: now 1790564686885525 us; error 62 us; skew 500 ppm
I20260928 03:04:46.885934 11769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:04:46.888576 11769 webserver.cc:460] Webserver started at http://127.6.192.2:36295/ using document root <none> and password file <none>
I20260928 03:04:46.889365 11769 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:46.889514 11769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:46.889853 11769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:46.892978 11769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "6481c7f7ab604c3a9790ef94a0ef6799"
format_stamp: "Formatted at 2026-09-28 03:04:46 on dist-test-slave-fjww"
server_key: "8da96b8d61cb8b2f0ba067dddc06e614"
server_key_iv: "2086c8045b72ba273aec7b030b8d3d40"
server_key_version: "encryptionkey@0"
I20260928 03:04:46.893949 11769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "6481c7f7ab604c3a9790ef94a0ef6799"
format_stamp: "Formatted at 2026-09-28 03:04:46 on dist-test-slave-fjww"
server_key: "8da96b8d61cb8b2f0ba067dddc06e614"
server_key_iv: "2086c8045b72ba273aec7b030b8d3d40"
server_key_version: "encryptionkey@0"
I20260928 03:04:46.899852 11769 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 03:04:46.903659 11783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:46.905186 11769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:04:46.905385 11769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "6481c7f7ab604c3a9790ef94a0ef6799"
format_stamp: "Formatted at 2026-09-28 03:04:46 on dist-test-slave-fjww"
server_key: "8da96b8d61cb8b2f0ba067dddc06e614"
server_key_iv: "2086c8045b72ba273aec7b030b8d3d40"
server_key_version: "encryptionkey@0"
I20260928 03:04:46.905683 11769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:46.911783 11769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:46.912659 11769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:46.912949 11769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:46.913911 11769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:46.915355 11769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:46.915493 11769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:46.915633 11769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:46.915720 11769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:46.950709 11769 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:44081
I20260928 03:04:46.950766 11895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:44081 every 8 connection(s)
I20260928 03:04:46.952243 11769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:04:46.956804 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 11769
I20260928 03:04:46.956984 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:04:46.957587 6912 external_mini_cluster.cc:1428] Setting key a78341a74be1a105218a4df7f62ccc3e
I20260928 03:04:46.961064 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:39213
--builtin_ntp_servers=127.6.192.20:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 03:04:46.962524 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:46.955029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:48461 (local address 127.6.192.62:39213)
0928 03:04:46.955244 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:46.955250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:46.957053 (+ 1803us) server_negotiation.cc:407] Connection header received
0928 03:04:46.959493 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:46.959498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:46.959580 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:46.959693 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:46.961799 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:46.961803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:46.961807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:04:46.961824 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:46.961859 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:04:46.961871 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:46.961976 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:46.962244 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 03:04:46.962325 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 03:04:46.964244 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:46.955655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:39213 (local address 127.6.192.2:48461)
0928 03:04:46.956841 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:46.956895 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:46.959087 (+ 2192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:46.960159 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:46.960173 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:46.960478 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:46.961351 (+ 873us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:04:46.961358 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:46.961434 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:04:46.961574 (+ 140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:46.961985 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:46.962000 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:46.962013 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 03:04:46.962360 (+ 347us) client_negotiation.cc:240] Negotiation successful
0928 03:04:46.962702 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":297,"threads_started":1}
I20260928 03:04:46.965896 11896 heartbeater.cc:347] Connected to a master server at 127.6.192.62:39213
I20260928 03:04:46.966259 11896 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:46.966909 11896 heartbeater.cc:511] Master 127.6.192.62:39213 requested a full tablet report, sending...
I20260928 03:04:46.968767 11579 ts_manager.cc:194] Registered new tserver with Master: 6481c7f7ab604c3a9790ef94a0ef6799 (127.6.192.2:44081)
W20260928 03:04:47.299432 11899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:47.299813 11899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:47.299906 11899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:47.307242 11899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:47.307354 11899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:47.307533 11899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:04:47.317093 11899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:39213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:47.318337 11899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:47.320156 11899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:47.333868 11904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:47.333911 11905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:47.333890 11907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:47.334846 11899 server_base.cc:1034] running on GCE node
I20260928 03:04:47.335547 11899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:47.336589 11899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:47.337908 11899 hybrid_clock.cc:648] HybridClock initialized: now 1790564687337828 us; error 106 us; skew 500 ppm
I20260928 03:04:47.338356 11899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:04:47.340914 11899 webserver.cc:460] Webserver started at http://127.6.192.3:36361/ using document root <none> and password file <none>
I20260928 03:04:47.341682 11899 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:47.341831 11899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:47.342191 11899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:47.345533 11899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "51b2ea81afb74fdb97521fe1bcd9f8de"
format_stamp: "Formatted at 2026-09-28 03:04:47 on dist-test-slave-fjww"
server_key: "44bff06417ba24a2569b57ef342d4b23"
server_key_iv: "8dbae29b646f24b52f58bbefa4659e17"
server_key_version: "encryptionkey@0"
I20260928 03:04:47.346360 11899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "51b2ea81afb74fdb97521fe1bcd9f8de"
format_stamp: "Formatted at 2026-09-28 03:04:47 on dist-test-slave-fjww"
server_key: "44bff06417ba24a2569b57ef342d4b23"
server_key_iv: "8dbae29b646f24b52f58bbefa4659e17"
server_key_version: "encryptionkey@0"
I20260928 03:04:47.351847 11899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 03:04:47.355814 11913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:47.357573 11899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:04:47.357800 11899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "51b2ea81afb74fdb97521fe1bcd9f8de"
format_stamp: "Formatted at 2026-09-28 03:04:47 on dist-test-slave-fjww"
server_key: "44bff06417ba24a2569b57ef342d4b23"
server_key_iv: "8dbae29b646f24b52f58bbefa4659e17"
server_key_version: "encryptionkey@0"
I20260928 03:04:47.358084 11899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:47.364703 11899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:47.365630 11899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:47.365944 11899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:47.366978 11899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:47.368721 11899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:47.368850 11899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:47.368991 11899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:47.369082 11899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:47.406883 11899 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:41761
I20260928 03:04:47.406939 12025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:41761 every 8 connection(s)
I20260928 03:04:47.408372 11899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:04:47.414701 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 11899
I20260928 03:04:47.414881 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:04:47.415467 6912 external_mini_cluster.cc:1428] Setting key 6e95da4e3d900e887cb17dc51e076109
I20260928 03:04:47.418041 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.411627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:33725 (local address 127.6.192.62:39213)
0928 03:04:47.411844 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:47.411850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:47.413258 (+ 1408us) server_negotiation.cc:407] Connection header received
0928 03:04:47.415399 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:47.415403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:47.415547 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:47.415664 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:47.417221 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:47.417227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:47.417231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:04:47.417253 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:47.417300 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:04:47.417317 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:47.417488 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:47.417694 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 03:04:47.417767 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 03:04:47.419109 12028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.412173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:39213 (local address 127.6.192.3:33725)
0928 03:04:47.413099 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:47.413140 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:47.415044 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:47.415942 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:47.415955 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:47.416095 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:47.416873 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:04:47.416877 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:47.416939 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:04:47.417045 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:47.417427 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:47.417440 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:47.417448 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 03:04:47.417773 (+ 325us) client_negotiation.cc:240] Negotiation successful
0928 03:04:47.418057 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":294,"threads_started":1}
I20260928 03:04:47.420437 12026 heartbeater.cc:347] Connected to a master server at 127.6.192.62:39213
I20260928 03:04:47.420744 12026 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:47.421340 12026 heartbeater.cc:511] Master 127.6.192.62:39213 requested a full tablet report, sending...
I20260928 03:04:47.422770 11579 ts_manager.cc:194] Registered new tserver with Master: 51b2ea81afb74fdb97521fe1bcd9f8de (127.6.192.3:41761)
I20260928 03:04:47.431860 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:04:47.440160 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.436912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.6.192.62:39213)
0928 03:04:47.437166 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:47.437173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:47.437825 (+ 652us) server_negotiation.cc:407] Connection header received
0928 03:04:47.437966 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:47.437970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:47.438025 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:47.438116 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:47.439432 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:47.439465 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:47.439469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:04:47.439487 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:47.439522 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:04:47.439535 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:47.439673 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:47.439825 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 03:04:47.439886 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 03:04:47.445161 11579 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:04:47.452488 11579 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:04:47.473323 12040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.467942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:41761 (local address 127.0.0.1:45324)
0928 03:04:47.468515 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:47.468533 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:47.468626 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:47.470476 (+ 1850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:47.470483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:47.470508 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:47.471279 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:04:47.471283 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:47.471314 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:04:47.471427 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:47.472583 (+ 1156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:47.472587 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:47.472592 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 03:04:47.472724 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 03:04:47.472778 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":267,"threads_started":1}
I20260928 03:04:47.473393 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.466462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.6.192.2:44081)
0928 03:04:47.467670 (+ 1208us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:47.467677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:47.467702 (+ 25us) server_negotiation.cc:407] Connection header received
0928 03:04:47.467775 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:47.467781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:47.467977 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:47.468185 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:47.471683 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:47.471695 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:47.471716 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:04:47.471782 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:47.471853 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:04:47.471882 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:47.472021 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:47.472925 (+ 904us) server_negotiation.cc:299] Negotiation successful
0928 03:04:47.473044 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1038,"thread_start_us":869,"threads_started":1}
I20260928 03:04:47.473842 12042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.468550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41019 (local address 127.0.0.1:55796)
0928 03:04:47.469064 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:47.469082 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:47.469165 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:47.471833 (+ 2668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:47.471838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:47.471860 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:47.472534 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:04:47.472539 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:47.472557 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:04:47.472598 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:47.473182 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:47.473187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:47.473193 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 03:04:47.473347 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 03:04:47.473396 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":722,"threads_started":1}
I20260928 03:04:47.474284 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.469108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55796 (local address 127.6.192.1:41019)
0928 03:04:47.471096 (+ 1988us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:47.471120 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:47.471166 (+ 46us) server_negotiation.cc:407] Connection header received
0928 03:04:47.471245 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:47.471255 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:47.471651 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:47.471834 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:47.472870 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:47.472878 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:47.472890 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:04:47.472917 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:47.472973 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:04:47.472996 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:47.473751 (+ 755us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:47.473842 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 03:04:47.473951 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1800,"thread_start_us":1655,"threads_started":1}
I20260928 03:04:47.474481 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.466256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:44081 (local address 127.0.0.1:59892)
0928 03:04:47.467103 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:47.467161 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:47.467333 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:47.469913 (+ 2580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:47.469921 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:47.470023 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:47.471314 (+ 1291us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:04:47.471318 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:47.471338 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:04:47.471426 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:47.472085 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:47.472097 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:47.472105 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 03:04:47.472650 (+ 545us) client_negotiation.cc:240] Negotiation successful
0928 03:04:47.472765 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":794,"threads_started":1}
I20260928 03:04:47.475085 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.468321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45324 (local address 127.6.192.3:41761)
0928 03:04:47.469774 (+ 1453us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:47.469786 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:47.469845 (+ 59us) server_negotiation.cc:407] Connection header received
0928 03:04:47.469937 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:47.469948 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:47.470243 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:47.470946 (+ 703us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:47.472297 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:47.472302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:47.472340 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:04:47.472362 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:47.472412 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:04:47.472450 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:47.474444 (+ 1994us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:47.474529 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 03:04:47.474640 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1244,"thread_start_us":1040,"threads_started":1}
I20260928 03:04:47.478375 11700 tablet_service.cc:1467] Processing CreateTablet for tablet b189958913bc4e1cb1c068a6b0210d17 (DEFAULT_TABLE table=test-table [id=662f85d774ee47a8a362ed19b083eeb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:47.478379 11961 tablet_service.cc:1467] Processing CreateTablet for tablet b189958913bc4e1cb1c068a6b0210d17 (DEFAULT_TABLE table=test-table [id=662f85d774ee47a8a362ed19b083eeb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:47.478379 11831 tablet_service.cc:1467] Processing CreateTablet for tablet b189958913bc4e1cb1c068a6b0210d17 (DEFAULT_TABLE table=test-table [id=662f85d774ee47a8a362ed19b083eeb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:04:47.479893 11831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b189958913bc4e1cb1c068a6b0210d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:47.479913 11700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b189958913bc4e1cb1c068a6b0210d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:47.479911 11961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b189958913bc4e1cb1c068a6b0210d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:47.500337 12046 tablet_bootstrap.cc:492] T b189958913bc4e1cb1c068a6b0210d17 P 4712fa96315c49ae84f3e000484c875b: Bootstrap starting.
I20260928 03:04:47.501278 12044 tablet_bootstrap.cc:492] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799: Bootstrap starting.
I20260928 03:04:47.501741 12045 tablet_bootstrap.cc:492] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de: Bootstrap starting.
I20260928 03:04:47.504640 11765 heartbeater.cc:503] Master 127.6.192.62:39213 was elected leader, sending a full tablet report...
I20260928 03:04:47.505398 12045 tablet_bootstrap.cc:654] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:47.505931 12046 tablet_bootstrap.cc:654] T b189958913bc4e1cb1c068a6b0210d17 P 4712fa96315c49ae84f3e000484c875b: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:47.506106 12044 tablet_bootstrap.cc:654] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:47.506687 12045 log.cc:824] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:47.507237 12046 log.cc:824] T b189958913bc4e1cb1c068a6b0210d17 P 4712fa96315c49ae84f3e000484c875b: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:47.507390 12044 log.cc:824] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:47.516904 12044 tablet_bootstrap.cc:492] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799: No bootstrap required, opened a new log
I20260928 03:04:47.516896 12045 tablet_bootstrap.cc:492] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de: No bootstrap required, opened a new log
I20260928 03:04:47.517233 12044 ts_tablet_manager.cc:1397] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20260928 03:04:47.517246 12045 ts_tablet_manager.cc:1397] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.005s
I20260928 03:04:47.517383 12046 tablet_bootstrap.cc:492] T b189958913bc4e1cb1c068a6b0210d17 P 4712fa96315c49ae84f3e000484c875b: No bootstrap required, opened a new log
I20260928 03:04:47.517673 12046 ts_tablet_manager.cc:1397] T b189958913bc4e1cb1c068a6b0210d17 P 4712fa96315c49ae84f3e000484c875b: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.008s
I20260928 03:04:47.524652 12044 raft_consensus.cc:348] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6481c7f7ab604c3a9790ef94a0ef6799" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 44081 } } peers { permanent_uuid: "51b2ea81afb74fdb97521fe1bcd9f8de" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41761 } } peers { permanent_uuid: "4712fa96315c49ae84f3e000484c875b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41019 } }
I20260928 03:04:47.525244 12044 raft_consensus.cc:374] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:47.525431 12044 raft_consensus.cc:729] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6481c7f7ab604c3a9790ef94a0ef6799, State: Initialized, Role: FOLLOWER
I20260928 03:04:47.526157 12044 consensus_queue.cc:260] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6481c7f7ab604c3a9790ef94a0ef6799" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 44081 } } peers { permanent_uuid: "51b2ea81afb74fdb97521fe1bcd9f8de" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41761 } } peers { permanent_uuid: "4712fa96315c49ae84f3e000484c875b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41019 } }
I20260928 03:04:47.527355 11896 heartbeater.cc:503] Master 127.6.192.62:39213 was elected leader, sending a full tablet report...
I20260928 03:04:47.528151 12044 ts_tablet_manager.cc:1428] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260928 03:04:47.529072 12046 raft_consensus.cc:348] T b189958913bc4e1cb1c068a6b0210d17 P 4712fa96315c49ae84f3e000484c875b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6481c7f7ab604c3a9790ef94a0ef6799" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 44081 } } peers { permanent_uuid: "51b2ea81afb74fdb97521fe1bcd9f8de" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41761 } } peers { permanent_uuid: "4712fa96315c49ae84f3e000484c875b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41019 } }
I20260928 03:04:47.529551 12046 raft_consensus.cc:374] T b189958913bc4e1cb1c068a6b0210d17 P 4712fa96315c49ae84f3e000484c875b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:47.529649 12046 raft_consensus.cc:729] T b189958913bc4e1cb1c068a6b0210d17 P 4712fa96315c49ae84f3e000484c875b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712fa96315c49ae84f3e000484c875b, State: Initialized, Role: FOLLOWER
I20260928 03:04:47.529428 12045 raft_consensus.cc:348] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6481c7f7ab604c3a9790ef94a0ef6799" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 44081 } } peers { permanent_uuid: "51b2ea81afb74fdb97521fe1bcd9f8de" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41761 } } peers { permanent_uuid: "4712fa96315c49ae84f3e000484c875b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41019 } }
I20260928 03:04:47.529872 12045 raft_consensus.cc:374] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:47.530016 12045 raft_consensus.cc:729] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b2ea81afb74fdb97521fe1bcd9f8de, State: Initialized, Role: FOLLOWER
I20260928 03:04:47.530318 12046 consensus_queue.cc:260] T b189958913bc4e1cb1c068a6b0210d17 P 4712fa96315c49ae84f3e000484c875b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6481c7f7ab604c3a9790ef94a0ef6799" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 44081 } } peers { permanent_uuid: "51b2ea81afb74fdb97521fe1bcd9f8de" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41761 } } peers { permanent_uuid: "4712fa96315c49ae84f3e000484c875b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41019 } }
I20260928 03:04:47.530607 12045 consensus_queue.cc:260] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6481c7f7ab604c3a9790ef94a0ef6799" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 44081 } } peers { permanent_uuid: "51b2ea81afb74fdb97521fe1bcd9f8de" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41761 } } peers { permanent_uuid: "4712fa96315c49ae84f3e000484c875b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41019 } }
I20260928 03:04:47.532439 12046 ts_tablet_manager.cc:1428] T b189958913bc4e1cb1c068a6b0210d17 P 4712fa96315c49ae84f3e000484c875b: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260928 03:04:47.533244 12026 heartbeater.cc:503] Master 127.6.192.62:39213 was elected leader, sending a full tablet report...
I20260928 03:04:47.534005 12045 ts_tablet_manager.cc:1428] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
I20260928 03:04:47.628383 12052 raft_consensus.cc:482] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:04:47.628774 12052 raft_consensus.cc:504] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6481c7f7ab604c3a9790ef94a0ef6799" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 44081 } } peers { permanent_uuid: "51b2ea81afb74fdb97521fe1bcd9f8de" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41761 } } peers { permanent_uuid: "4712fa96315c49ae84f3e000484c875b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41019 } }
I20260928 03:04:47.630813 12052 leader_election.cc:288] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6481c7f7ab604c3a9790ef94a0ef6799 (127.6.192.2:44081), 4712fa96315c49ae84f3e000484c875b (127.6.192.1:41019)
I20260928 03:04:47.633787 12028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.631231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:44081 (local address 127.6.192.3:43401)
0928 03:04:47.631509 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:47.631537 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:47.631663 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:47.632065 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:47.632070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:47.632101 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:47.632874 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:04:47.632877 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:47.632894 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:04:47.632941 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:47.633334 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:47.633340 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:47.633345 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 03:04:47.633469 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 03:04:47.633520 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 03:04:47.634164 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.631216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:43401 (local address 127.6.192.2:44081)
0928 03:04:47.631424 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:47.631432 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:47.631619 (+ 187us) server_negotiation.cc:407] Connection header received
0928 03:04:47.631805 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:47.631813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:47.631890 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:47.632009 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:47.633115 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:47.633122 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:47.633126 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:04:47.633149 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:47.633190 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:04:47.633209 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:47.633327 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:47.633772 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 03:04:47.633827 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 03:04:47.634846 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.632058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:36353 (local address 127.6.192.1:41019)
0928 03:04:47.632265 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:47.632273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:47.632730 (+ 457us) server_negotiation.cc:407] Connection header received
0928 03:04:47.632874 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:47.632879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:47.632931 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:47.633038 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:47.634029 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:47.634036 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:47.634040 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:04:47.634061 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:47.634100 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:04:47.634120 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:47.634221 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:47.634464 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 03:04:47.634557 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 03:04:47.634930 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.631739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41019 (local address 127.6.192.3:36353)
0928 03:04:47.632627 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:47.632641 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:47.632725 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:47.633075 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:47.633081 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:47.633106 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:47.633779 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:04:47.633783 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:47.633801 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:04:47.633856 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:47.634317 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:47.634321 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:47.634326 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 03:04:47.634432 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 03:04:47.634477 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":611,"threads_started":1}
I20260928 03:04:47.635255 11851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b189958913bc4e1cb1c068a6b0210d17" candidate_uuid: "51b2ea81afb74fdb97521fe1bcd9f8de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6481c7f7ab604c3a9790ef94a0ef6799" is_pre_election: true
I20260928 03:04:47.635793 11851 raft_consensus.cc:2454] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b2ea81afb74fdb97521fe1bcd9f8de in term 0.
I20260928 03:04:47.636482 11915 leader_election.cc:302] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b2ea81afb74fdb97521fe1bcd9f8de, 6481c7f7ab604c3a9790ef94a0ef6799; no voters:
I20260928 03:04:47.636667 11720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b189958913bc4e1cb1c068a6b0210d17" candidate_uuid: "51b2ea81afb74fdb97521fe1bcd9f8de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712fa96315c49ae84f3e000484c875b" is_pre_election: true
I20260928 03:04:47.636884 12052 raft_consensus.cc:2781] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:04:47.637019 12052 raft_consensus.cc:482] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:04:47.637142 12052 raft_consensus.cc:3037] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:47.637315 11720 raft_consensus.cc:2454] T b189958913bc4e1cb1c068a6b0210d17 P 4712fa96315c49ae84f3e000484c875b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b2ea81afb74fdb97521fe1bcd9f8de in term 0.
I20260928 03:04:47.639671 12052 raft_consensus.cc:504] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6481c7f7ab604c3a9790ef94a0ef6799" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 44081 } } peers { permanent_uuid: "51b2ea81afb74fdb97521fe1bcd9f8de" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41761 } } peers { permanent_uuid: "4712fa96315c49ae84f3e000484c875b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41019 } }
I20260928 03:04:47.640302 12052 leader_election.cc:288] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [CANDIDATE]: Term 1 election: Requested vote from peers 6481c7f7ab604c3a9790ef94a0ef6799 (127.6.192.2:44081), 4712fa96315c49ae84f3e000484c875b (127.6.192.1:41019)
I20260928 03:04:47.640900 11851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b189958913bc4e1cb1c068a6b0210d17" candidate_uuid: "51b2ea81afb74fdb97521fe1bcd9f8de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6481c7f7ab604c3a9790ef94a0ef6799"
I20260928 03:04:47.640897 11720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b189958913bc4e1cb1c068a6b0210d17" candidate_uuid: "51b2ea81afb74fdb97521fe1bcd9f8de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712fa96315c49ae84f3e000484c875b"
I20260928 03:04:47.641146 11720 raft_consensus.cc:3037] T b189958913bc4e1cb1c068a6b0210d17 P 4712fa96315c49ae84f3e000484c875b [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:47.641237 11851 raft_consensus.cc:3037] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:47.644891 11720 raft_consensus.cc:2454] T b189958913bc4e1cb1c068a6b0210d17 P 4712fa96315c49ae84f3e000484c875b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b2ea81afb74fdb97521fe1bcd9f8de in term 1.
I20260928 03:04:47.644896 11851 raft_consensus.cc:2454] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b2ea81afb74fdb97521fe1bcd9f8de in term 1.
I20260928 03:04:47.645443 11915 leader_election.cc:302] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b2ea81afb74fdb97521fe1bcd9f8de, 6481c7f7ab604c3a9790ef94a0ef6799; no voters:
I20260928 03:04:47.645823 12052 raft_consensus.cc:2781] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:47.646350 12052 raft_consensus.cc:686] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [term 1 LEADER]: Becoming Leader. State: Replica: 51b2ea81afb74fdb97521fe1bcd9f8de, State: Running, Role: LEADER
I20260928 03:04:47.646872 12052 consensus_queue.cc:237] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6481c7f7ab604c3a9790ef94a0ef6799" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 44081 } } peers { permanent_uuid: "51b2ea81afb74fdb97521fe1bcd9f8de" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41761 } } peers { permanent_uuid: "4712fa96315c49ae84f3e000484c875b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41019 } }
I20260928 03:04:47.651628 11579 catalog_manager.cc:5495] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de reported cstate change: term changed from 0 to 1, leader changed from <none> to 51b2ea81afb74fdb97521fe1bcd9f8de (127.6.192.3). New cstate: current_term: 1 leader_uuid: "51b2ea81afb74fdb97521fe1bcd9f8de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6481c7f7ab604c3a9790ef94a0ef6799" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 44081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b2ea81afb74fdb97521fe1bcd9f8de" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4712fa96315c49ae84f3e000484c875b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41019 } health_report { overall_health: UNKNOWN } } }
W20260928 03:04:47.660274 12027 tablet.cc:2369] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:04:47.676069 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.673851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45332 (local address 127.6.192.3:41761)
0928 03:04:47.674026 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:47.674030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:47.674195 (+ 165us) server_negotiation.cc:407] Connection header received
0928 03:04:47.674389 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:47.674393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:47.674474 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:47.674566 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:47.675397 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:47.675402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:47.675406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:04:47.675424 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:47.675524 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:04:47.675540 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:47.675674 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:47.675800 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 03:04:47.675856 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 03:04:47.685038 11720 raft_consensus.cc:1261] T b189958913bc4e1cb1c068a6b0210d17 P 4712fa96315c49ae84f3e000484c875b [term 1 FOLLOWER]: Refusing update from remote peer 51b2ea81afb74fdb97521fe1bcd9f8de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:47.685078 11851 raft_consensus.cc:1261] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799 [term 1 FOLLOWER]: Refusing update from remote peer 51b2ea81afb74fdb97521fe1bcd9f8de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:04:47.686159 12054 consensus_queue.cc:1035] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712fa96315c49ae84f3e000484c875b" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:04:47.686638 12052 consensus_queue.cc:1035] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [LEADER]: Connected to new peer: Peer: permanent_uuid: "6481c7f7ab604c3a9790ef94a0ef6799" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 44081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 03:04:47.704351 11897 tablet.cc:2369] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:04:47.716777 11578 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:04:47.717094 11578 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:04:47.721002 11578 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44c0ab683445499d9f14f7f240616d25: Sending DeleteTablet for 3 replicas of tablet b189958913bc4e1cb1c068a6b0210d17
I20260928 03:04:47.722281 11831 tablet_service.cc:1514] Processing DeleteTablet for tablet b189958913bc4e1cb1c068a6b0210d17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:47 UTC) from {username='slave'} at 127.0.0.1:59892
I20260928 03:04:47.722486 11700 tablet_service.cc:1514] Processing DeleteTablet for tablet b189958913bc4e1cb1c068a6b0210d17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:47 UTC) from {username='slave'} at 127.0.0.1:55796
I20260928 03:04:47.722790 11961 tablet_service.cc:1514] Processing DeleteTablet for tablet b189958913bc4e1cb1c068a6b0210d17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:04:47 UTC) from {username='slave'} at 127.0.0.1:45324
I20260928 03:04:47.723510 12067 tablet_replica.cc:330] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799: stopping tablet replica
I20260928 03:04:47.724133 12067 raft_consensus.cc:2229] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:04:47.724700 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 11638
I20260928 03:04:47.724831 12067 raft_consensus.cc:2258] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:47.731107 12067 ts_tablet_manager.cc:1905] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:47.739629 12068 tablet_replica.cc:330] T b189958913bc4e1cb1c068a6b0210d17 P 4712fa96315c49ae84f3e000484c875b: stopping tablet replica
I20260928 03:04:47.739889 12069 tablet_replica.cc:330] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de: stopping tablet replica
I20260928 03:04:47.740129 12068 raft_consensus.cc:2229] T b189958913bc4e1cb1c068a6b0210d17 P 4712fa96315c49ae84f3e000484c875b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:04:47.740378 12069 raft_consensus.cc:2229] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:04:47.741256 12069 raft_consensus.cc:2258] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:47.741653 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.727350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55806 (local address 127.6.192.1:41019)
0928 03:04:47.727618 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:47.727627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:47.737610 (+ 9983us) server_negotiation.cc:407] Connection header received
0928 03:04:47.738165 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:47.738171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:47.738297 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:47.738389 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:47.740736 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:47.740741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:47.740745 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:04:47.740775 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:47.740819 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:04:47.740837 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:47.740946 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:47.741255 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 03:04:47.741362 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260928 03:04:47.742681 12068 raft_consensus.cc:2258] T b189958913bc4e1cb1c068a6b0210d17 P 4712fa96315c49ae84f3e000484c875b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:04:47.742985 11760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:47.744724 12069 ts_tablet_manager.cc:1905] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:04:47.746899 12067 ts_tablet_manager.cc:1918] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:04:47.747133 12067 log.cc:1196] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/b189958913bc4e1cb1c068a6b0210d17
I20260928 03:04:47.747785 12067 ts_tablet_manager.cc:1939] T b189958913bc4e1cb1c068a6b0210d17 P 6481c7f7ab604c3a9790ef94a0ef6799: Deleting consensus metadata
I20260928 03:04:47.749897 11564 catalog_manager.cc:4841] TS 6481c7f7ab604c3a9790ef94a0ef6799 (127.6.192.2:44081): tablet b189958913bc4e1cb1c068a6b0210d17 (table test-table [id=662f85d774ee47a8a362ed19b083eeb0]) successfully deleted
I20260928 03:04:47.753070 12069 ts_tablet_manager.cc:1918] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:04:47.753307 12069 log.cc:1196] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/b189958913bc4e1cb1c068a6b0210d17
I20260928 03:04:47.753832 12069 ts_tablet_manager.cc:1939] T b189958913bc4e1cb1c068a6b0210d17 P 51b2ea81afb74fdb97521fe1bcd9f8de: Deleting consensus metadata
I20260928 03:04:47.755407 11564 catalog_manager.cc:4841] TS 51b2ea81afb74fdb97521fe1bcd9f8de (127.6.192.3:41761): tablet b189958913bc4e1cb1c068a6b0210d17 (table test-table [id=662f85d774ee47a8a362ed19b083eeb0]) successfully deleted
I20260928 03:04:47.858204 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 11638
I20260928 03:04:47.875682 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 11769
W20260928 03:04:47.875725 11564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.192.1:41019 (error 108)
I20260928 03:04:47.878063 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.877563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41019 (local address 127.0.0.1:55808)
0928 03:04:47.877733 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:47.877889 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 03:04:47.878304 11564 catalog_manager.cc:4584] TS 4712fa96315c49ae84f3e000484c875b (127.6.192.1:41019): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b189958913bc4e1cb1c068a6b0210d17: Network error: Client connection negotiation failed: client connection to 127.6.192.1:41019: connect: Connection refused (error 111)
I20260928 03:04:47.879601 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.877218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.6.192.2:44081)
0928 03:04:47.877418 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:47.877424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:47.877446 (+ 22us) server_negotiation.cc:407] Connection header received
0928 03:04:47.877514 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:47.877519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:47.877630 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:47.877743 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:47.878913 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:47.878919 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:47.878924 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:04:47.878945 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:47.878985 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:04:47.879001 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:47.879129 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:47.879227 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 03:04:47.879287 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 03:04:47.880306 11891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:47.932178 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.931547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41019 (local address 127.0.0.1:55820)
0928 03:04:47.931825 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:47.931959 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 03:04:47.933467 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.933028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41019 (local address 127.0.0.1:55824)
0928 03:04:47.933207 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:47.933301 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 03:04:47.976615 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 11769
I20260928 03:04:47.994138 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.993656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41019 (local address 127.0.0.1:55830)
0928 03:04:47.993908 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:47.993995 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 03:04:47.995188 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.994853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41019 (local address 127.0.0.1:55842)
0928 03:04:47.994998 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:47.995072 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 03:04:47.999069 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 11899
I20260928 03:04:48.002359 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:47.999929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45346 (local address 127.6.192.3:41761)
0928 03:04:48.000123 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:48.000129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:48.000152 (+ 23us) server_negotiation.cc:407] Connection header received
0928 03:04:48.000351 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:48.000357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:48.000427 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:48.000558 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:48.001583 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:48.001589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:48.001594 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:04:48.001613 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:48.001663 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:04:48.001679 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:48.001833 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:48.001989 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 03:04:48.002049 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 03:04:48.003212 12021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:48.076638 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:48.076111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41019 (local address 127.0.0.1:55854)
0928 03:04:48.076365 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:48.076472 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 03:04:48.077837 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:48.077424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41019 (local address 127.0.0.1:55866)
0928 03:04:48.077632 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:48.077709 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 03:04:48.122989 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 11899
I20260928 03:04:48.142228 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 11548
I20260928 03:04:48.143530 11609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:48.256364 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:48.255654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41019 (local address 127.0.0.1:55870)
0928 03:04:48.256009 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:48.256137 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260928 03:04:48.256582 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 11548
2026-09-28T03:04:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2873 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 03:04:48.298516 6912 mini_oidc.cc:152] Starting JWKS server
I20260928 03:04:48.301239 6912 webserver.cc:460] Webserver started at https://127.0.0.1:43195/ using document root <none> and password file <none>
I20260928 03:04:48.301579 6912 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 03:04:48.302390 6912 webserver.cc:460] Webserver started at http://127.0.0.1:45917/ using document root <none> and password file <none>
2026-09-28T03:04:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:04:48Z Disabled control of system clock
I20260928 03:04:48.333153 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:39065
--webserver_interface=127.6.192.62
--webserver_port=0
--builtin_ntp_servers=127.6.192.20:39571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:39065
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43195/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0/testchainca.cert with env {}
W20260928 03:04:48.675892 12081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:48.676209 12081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:48.676321 12081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:48.684033 12081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:04:48.684157 12081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:48.684197 12081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:48.684267 12081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:04:48.684312 12081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 03:04:48.684368 12081 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 03:04:48.684401 12081 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43195/jwks.json
I20260928 03:04:48.695544 12081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:39571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:39065
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:39065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43195/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:48.696892 12081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:48.698274 12081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:48.711014 12086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:48.710979 12089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:48.710911 12087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:48.711800 12081 server_base.cc:1034] running on GCE node
I20260928 03:04:48.712646 12081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:48.714133 12081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:48.715402 12081 hybrid_clock.cc:648] HybridClock initialized: now 1790564688715355 us; error 37 us; skew 500 ppm
I20260928 03:04:48.715808 12081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:04:48.718501 12081 webserver.cc:460] Webserver started at http://127.6.192.62:38385/ using document root <none> and password file <none>
I20260928 03:04:48.719254 12081 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:48.719398 12081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:48.719745 12081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:48.722797 12081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "15de73b7f447445b995a331b1988152e"
format_stamp: "Formatted at 2026-09-28 03:04:48 on dist-test-slave-fjww"
server_key: "8755fae6a0593d0fc210a05024cc43ce"
server_key_iv: "b76f82672adcd0b3d4146c252ba9bba0"
server_key_version: "encryptionkey@0"
I20260928 03:04:48.723728 12081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "15de73b7f447445b995a331b1988152e"
format_stamp: "Formatted at 2026-09-28 03:04:48 on dist-test-slave-fjww"
server_key: "8755fae6a0593d0fc210a05024cc43ce"
server_key_iv: "b76f82672adcd0b3d4146c252ba9bba0"
server_key_version: "encryptionkey@0"
I20260928 03:04:48.729341 12081 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 03:04:48.733496 12095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:48.735082 12081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 03:04:48.735348 12081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "15de73b7f447445b995a331b1988152e"
format_stamp: "Formatted at 2026-09-28 03:04:48 on dist-test-slave-fjww"
server_key: "8755fae6a0593d0fc210a05024cc43ce"
server_key_iv: "b76f82672adcd0b3d4146c252ba9bba0"
server_key_version: "encryptionkey@0"
I20260928 03:04:48.735671 12081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:48.755183 12081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:48.756260 12081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:48.756616 12081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:48.776662 12081 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:39065
I20260928 03:04:48.776706 12146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:39065 every 8 connection(s)
I20260928 03:04:48.778142 12081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:04:48.782547 12147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:48.785254 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 12081
I20260928 03:04:48.785517 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:04:48.786166 6912 external_mini_cluster.cc:1428] Setting key ad7fd0cc8a731725e83a8a7a0ee669e4
I20260928 03:04:48.795255 12147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e: Bootstrap starting.
I20260928 03:04:48.800479 12147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:48.802099 12147 log.cc:824] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:48.804234 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:48.788422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.6.192.62:39065)
0928 03:04:48.789632 (+ 1210us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:48.789646 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:48.789695 (+ 49us) server_negotiation.cc:407] Connection header received
0928 03:04:48.791748 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:48.791776 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:48.792418 (+ 642us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:48.792967 (+ 549us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:48.793673 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:48.795131 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:48.797722 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:48.798816 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:48.799040 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:48.800871 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:48.800883 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:48.800910 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:04:48.801760 (+ 850us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:48.801914 (+ 154us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:04:48.801956 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:48.802097 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:48.802428 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 03:04:48.802821 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":443,"threads_started":1}
I20260928 03:04:48.806172 12147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e: No bootstrap required, opened a new log
I20260928 03:04:48.808799 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:04:48.814739 12147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15de73b7f447445b995a331b1988152e" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 39065 } }
I20260928 03:04:48.815207 12147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:48.815325 12147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15de73b7f447445b995a331b1988152e, State: Initialized, Role: FOLLOWER
I20260928 03:04:48.816051 12147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15de73b7f447445b995a331b1988152e" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 39065 } }
I20260928 03:04:48.816318 12147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:04:48.816465 12147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:04:48.816648 12147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:48.819178 12147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15de73b7f447445b995a331b1988152e" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 39065 } }
I20260928 03:04:48.819808 12147 leader_election.cc:302] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15de73b7f447445b995a331b1988152e; no voters:
I20260928 03:04:48.820441 12147 leader_election.cc:288] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:04:48.820619 12152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:48.821390 12152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e [term 1 LEADER]: Becoming Leader. State: Replica: 15de73b7f447445b995a331b1988152e, State: Running, Role: LEADER
I20260928 03:04:48.822160 12152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15de73b7f447445b995a331b1988152e" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 39065 } }
I20260928 03:04:48.822990 12147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:04:48.826056 12153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15de73b7f447445b995a331b1988152e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15de73b7f447445b995a331b1988152e" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 39065 } } }
I20260928 03:04:48.826439 12154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15de73b7f447445b995a331b1988152e. Latest consensus state: current_term: 1 leader_uuid: "15de73b7f447445b995a331b1988152e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15de73b7f447445b995a331b1988152e" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 39065 } } }
I20260928 03:04:48.826685 12153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:48.826757 12154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:48.828490 12159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:04:48.834152 12159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:04:48.844578 12159 catalog_manager.cc:1347] Generated new cluster ID: ffb5d50b9e504597a99b861d240e3df6
I20260928 03:04:48.844769 12159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:04:48.864331 12159 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:04:48.865854 12159 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:04:48.881289 12159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15de73b7f447445b995a331b1988152e: Generated new TSK 0
I20260928 03:04:48.882344 12159 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 03:04:48.929884 12177 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 03:04:48.930083 12177 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 03:04:48.931094 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:48.906075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59234 (local address 127.6.192.62:39065)
0928 03:04:48.906434 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:48.906444 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:48.906912 (+ 468us) server_negotiation.cc:407] Connection header received
0928 03:04:48.907309 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:48.907317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:48.907400 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:48.907641 (+ 241us) server_negotiation.cc:226] Negotiated authn=JWT
0928 03:04:48.908346 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:48.909216 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:48.911095 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:48.911691 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:48.911842 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:48.912464 (+ 622us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 03:04:48.930524 (+ 18060us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43195/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 03:04:48.930849 (+ 325us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59234: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43195/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 03:04:48.931257 12150 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59234: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43195/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 03:04:48.933218 12176 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:04:48.905998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:39065 (local address 127.0.0.1:59234)
0928 03:04:48.906773 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:48.906797 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:48.907031 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:48.907738 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:48.907744 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:48.907759 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0928 03:04:48.908138 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:48.908154 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:48.909396 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:48.909405 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:48.910920 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:48.910934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:48.911902 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:48.911910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:48.912070 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:48.912139 (+ 69us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 03:04:48.930886 (+ 18747us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43195/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 03:04:48.931302 (+ 416us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.192.62:39065: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43195/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":415,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.6.192.62:39065: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43195/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 03:04:48.935882 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 12081
I20260928 03:04:48.937322 12142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:49.041160 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 12081
2026-09-28T03:04:49Z chronyd exiting
I20260928 03:04:49.093675 6912 test_util.cc:175] -----------------------------------------------
I20260928 03:04:49.093828 6912 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790564644636890-6912-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (799 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 03:04:49.097671 6912 mini_oidc.cc:152] Starting JWKS server
I20260928 03:04:49.099684 6912 webserver.cc:460] Webserver started at https://127.0.0.1:38007/ using document root <none> and password file <none>
I20260928 03:04:49.099932 6912 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 03:04:49.100654 6912 webserver.cc:460] Webserver started at http://127.0.0.1:42137/ using document root <none> and password file <none>
2026-09-28T03:04:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:04:49Z Disabled control of system clock
I20260928 03:04:49.130053 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:38721
--webserver_interface=127.6.192.62
--webserver_port=0
--builtin_ntp_servers=127.6.192.20:46305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:38721
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38007/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564644636890-6912-0/testchainca.cert with env {}
W20260928 03:04:49.470357 12184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:49.470733 12184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:49.470916 12184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:49.478847 12184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:04:49.479030 12184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:49.479125 12184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:49.479242 12184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:04:49.479322 12184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 03:04:49.479411 12184 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 03:04:49.479514 12184 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38007/jwks.json
I20260928 03:04:49.490482 12184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:38721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:38721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38007/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564644636890-6912-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:49.491969 12184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:49.493403 12184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:49.506453 12189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:49.506475 12192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:49.506412 12190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:49.512645 12184 server_base.cc:1034] running on GCE node
I20260928 03:04:49.513638 12184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:49.515421 12184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:49.516806 12184 hybrid_clock.cc:648] HybridClock initialized: now 1790564689516736 us; error 91 us; skew 500 ppm
I20260928 03:04:49.517386 12184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:04:49.520459 12184 webserver.cc:460] Webserver started at http://127.6.192.62:34381/ using document root <none> and password file <none>
I20260928 03:04:49.521293 12184 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:49.521443 12184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:49.521788 12184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:49.524948 12184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "1de4777559554661a4a2e0865a9d3500"
format_stamp: "Formatted at 2026-09-28 03:04:49 on dist-test-slave-fjww"
server_key: "94a8ce552ea72165e8879a76efc44e5a"
server_key_iv: "f6538d6736cc9723b9f5320d3946979a"
server_key_version: "encryptionkey@0"
I20260928 03:04:49.526050 12184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "1de4777559554661a4a2e0865a9d3500"
format_stamp: "Formatted at 2026-09-28 03:04:49 on dist-test-slave-fjww"
server_key: "94a8ce552ea72165e8879a76efc44e5a"
server_key_iv: "f6538d6736cc9723b9f5320d3946979a"
server_key_version: "encryptionkey@0"
I20260928 03:04:49.532684 12184 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260928 03:04:49.536922 12198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:49.538758 12184 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:04:49.539008 12184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "1de4777559554661a4a2e0865a9d3500"
format_stamp: "Formatted at 2026-09-28 03:04:49 on dist-test-slave-fjww"
server_key: "94a8ce552ea72165e8879a76efc44e5a"
server_key_iv: "f6538d6736cc9723b9f5320d3946979a"
server_key_version: "encryptionkey@0"
I20260928 03:04:49.539417 12184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:49.565685 12184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:49.566730 12184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:49.567128 12184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:49.589717 12184 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:38721
I20260928 03:04:49.589749 12249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:38721 every 8 connection(s)
I20260928 03:04:49.591419 12184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:04:49.592715 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 12184
I20260928 03:04:49.592937 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:04:49.593729 6912 external_mini_cluster.cc:1428] Setting key be82e47f048d0b4fc2adb05cc5ee6470
I20260928 03:04:49.596843 12250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:49.607632 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:49.595857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.6.192.62:38721)
0928 03:04:49.596997 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:49.597010 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:49.597054 (+ 44us) server_negotiation.cc:407] Connection header received
0928 03:04:49.598685 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:49.598709 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:49.599172 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:49.599637 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:49.600222 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:49.601728 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:49.603048 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:49.603799 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:49.603988 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:49.605197 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:49.605211 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:49.605225 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:04:49.605830 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:49.605916 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:04:49.605940 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:49.606080 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:49.606319 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 03:04:49.606628 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":414,"threads_started":1}
I20260928 03:04:49.608191 12250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500: Bootstrap starting.
I20260928 03:04:49.611675 12250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:49.611716 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:04:49.612682 12250 log.cc:824] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:49.616003 12250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500: No bootstrap required, opened a new log
I20260928 03:04:49.623939 12250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de4777559554661a4a2e0865a9d3500" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38721 } }
I20260928 03:04:49.624439 12250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:49.624557 12250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1de4777559554661a4a2e0865a9d3500, State: Initialized, Role: FOLLOWER
I20260928 03:04:49.625200 12250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de4777559554661a4a2e0865a9d3500" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38721 } }
I20260928 03:04:49.625478 12250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:04:49.625633 12250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:04:49.625842 12250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:49.628262 12250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de4777559554661a4a2e0865a9d3500" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38721 } }
I20260928 03:04:49.628813 12250 leader_election.cc:302] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1de4777559554661a4a2e0865a9d3500; no voters:
I20260928 03:04:49.629500 12250 leader_election.cc:288] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:04:49.629714 12255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:49.630445 12255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500 [term 1 LEADER]: Becoming Leader. State: Replica: 1de4777559554661a4a2e0865a9d3500, State: Running, Role: LEADER
I20260928 03:04:49.631263 12255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de4777559554661a4a2e0865a9d3500" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38721 } }
I20260928 03:04:49.631862 12250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:04:49.634395 12257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1de4777559554661a4a2e0865a9d3500. Latest consensus state: current_term: 1 leader_uuid: "1de4777559554661a4a2e0865a9d3500" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de4777559554661a4a2e0865a9d3500" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38721 } } }
I20260928 03:04:49.634395 12256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1de4777559554661a4a2e0865a9d3500" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de4777559554661a4a2e0865a9d3500" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38721 } } }
I20260928 03:04:49.635236 12257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:49.635252 12256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:49.637374 12261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:04:49.643116 12261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:04:49.652923 12261 catalog_manager.cc:1347] Generated new cluster ID: 90aa656ef2b744c8b69c73c5daa1226f
I20260928 03:04:49.653067 12261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:04:49.670167 12261 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:04:49.671880 12261 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:04:49.689450 12261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1de4777559554661a4a2e0865a9d3500: Generated new TSK 0
I20260928 03:04:49.690388 12261 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 03:04:49.724102 12280 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 03:04:49.724242 12280 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 03:04:49.725400 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:49.704083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46590 (local address 127.6.192.62:38721)
0928 03:04:49.704351 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:49.704362 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:49.704888 (+ 526us) server_negotiation.cc:407] Connection header received
0928 03:04:49.705138 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:49.705142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:49.705312 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:49.705462 (+ 150us) server_negotiation.cc:226] Negotiated authn=JWT
0928 03:04:49.706024 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:49.706874 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:49.709101 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:49.709631 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:49.709754 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:49.710441 (+ 687us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 03:04:49.724799 (+ 14358us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 03:04:49.725135 (+ 336us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46590: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":80}
W20260928 03:04:49.725565 12253 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46590: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 03:04:49.725864 12279 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:04:49.704040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:38721 (local address 127.0.0.1:46590)
0928 03:04:49.704743 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:49.704765 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:49.704947 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:49.705509 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:49.705516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:49.705534 (+ 18us) client_negotiation.cc:189] Negotiated authn=JWT
0928 03:04:49.705859 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:49.705873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:49.707047 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:49.707053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:49.708885 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:49.708906 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:49.709823 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:49.709829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:49.710048 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:49.710099 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 03:04:49.725088 (+ 14989us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 03:04:49.725414 (+ 326us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.192.62:38721: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":393,"threads_started":1}
I20260928 03:04:49.727368 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 12184
I20260928 03:04:49.728888 12245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:49.825506 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 12184
2026-09-28T03:04:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (784 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 03:04:49.881915 6912 mini_oidc.cc:152] Starting JWKS server
I20260928 03:04:49.883935 6912 webserver.cc:460] Webserver started at https://127.0.0.1:46459/ using document root <none> and password file <none>
I20260928 03:04:49.884212 6912 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 03:04:49.884876 6912 webserver.cc:460] Webserver started at http://127.0.0.1:34821/ using document root <none> and password file <none>
2026-09-28T03:04:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:04:49Z Disabled control of system clock
I20260928 03:04:49.914767 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:43331
--webserver_interface=127.6.192.62
--webserver_port=0
--builtin_ntp_servers=127.6.192.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:43331
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46459/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 03:04:50.251158 12287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:50.251570 12287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:50.251731 12287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:50.259414 12287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:04:50.259575 12287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:50.259672 12287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:50.259770 12287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:04:50.259858 12287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 03:04:50.259948 12287 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 03:04:50.260000 12287 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46459/jwks.json
I20260928 03:04:50.269822 12287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:43331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:43331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46459/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:50.271296 12287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:50.273250 12287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:50.286552 12292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:50.286592 12293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:50.286518 12295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:50.287868 12287 server_base.cc:1034] running on GCE node
I20260928 03:04:50.288756 12287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:50.290582 12287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:50.291872 12287 hybrid_clock.cc:648] HybridClock initialized: now 1790564690291811 us; error 54 us; skew 500 ppm
I20260928 03:04:50.292424 12287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:04:50.301237 12287 webserver.cc:460] Webserver started at http://127.6.192.62:46297/ using document root <none> and password file <none>
I20260928 03:04:50.302132 12287 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:50.302266 12287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:50.302645 12287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:50.305886 12287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "73d745fbe3b1460bb77b6b88fa014e96"
format_stamp: "Formatted at 2026-09-28 03:04:50 on dist-test-slave-fjww"
server_key: "8ca636062a5db4f80f1d15d98c169938"
server_key_iv: "7f8b630e31efb94df990bd2c4124afaf"
server_key_version: "encryptionkey@0"
I20260928 03:04:50.306838 12287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "73d745fbe3b1460bb77b6b88fa014e96"
format_stamp: "Formatted at 2026-09-28 03:04:50 on dist-test-slave-fjww"
server_key: "8ca636062a5db4f80f1d15d98c169938"
server_key_iv: "7f8b630e31efb94df990bd2c4124afaf"
server_key_version: "encryptionkey@0"
I20260928 03:04:50.312783 12287 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 03:04:50.316919 12301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:50.318537 12287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 03:04:50.318758 12287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "73d745fbe3b1460bb77b6b88fa014e96"
format_stamp: "Formatted at 2026-09-28 03:04:50 on dist-test-slave-fjww"
server_key: "8ca636062a5db4f80f1d15d98c169938"
server_key_iv: "7f8b630e31efb94df990bd2c4124afaf"
server_key_version: "encryptionkey@0"
I20260928 03:04:50.319000 12287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:50.338059 12287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:50.339056 12287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:50.339385 12287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:50.359103 12287 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:43331
I20260928 03:04:50.359113 12352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:43331 every 8 connection(s)
I20260928 03:04:50.360610 12287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:04:50.364753 12353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:50.366261 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 12287
I20260928 03:04:50.366489 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:04:50.367362 6912 external_mini_cluster.cc:1428] Setting key a68c1c2c00779ed225373ff3a63cb312
I20260928 03:04:50.381860 12353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96: Bootstrap starting.
I20260928 03:04:50.384740 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:50.369717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50806 (local address 127.6.192.62:43331)
0928 03:04:50.370883 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:50.370898 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:50.370946 (+ 48us) server_negotiation.cc:407] Connection header received
0928 03:04:50.372601 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:50.372629 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:50.373188 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:50.373692 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:50.374587 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:50.376563 (+ 1976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:50.378796 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:50.380038 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:50.380254 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:50.381670 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:50.381681 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:50.381703 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:04:50.382517 (+ 814us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:50.382625 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:04:50.382661 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:50.382772 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:50.383211 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 03:04:50.383644 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":421,"threads_started":1}
I20260928 03:04:50.385996 12353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:50.387166 12353 log.cc:824] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:50.389329 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:04:50.390666 12353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96: No bootstrap required, opened a new log
I20260928 03:04:50.398722 12353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d745fbe3b1460bb77b6b88fa014e96" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43331 } }
I20260928 03:04:50.399245 12353 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:50.399334 12353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d745fbe3b1460bb77b6b88fa014e96, State: Initialized, Role: FOLLOWER
I20260928 03:04:50.400080 12353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d745fbe3b1460bb77b6b88fa014e96" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43331 } }
I20260928 03:04:50.400302 12353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:04:50.400454 12353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:04:50.400626 12353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:50.402858 12353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d745fbe3b1460bb77b6b88fa014e96" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43331 } }
I20260928 03:04:50.403429 12353 leader_election.cc:302] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73d745fbe3b1460bb77b6b88fa014e96; no voters:
I20260928 03:04:50.404167 12353 leader_election.cc:288] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:04:50.404302 12358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:50.405153 12358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96 [term 1 LEADER]: Becoming Leader. State: Replica: 73d745fbe3b1460bb77b6b88fa014e96, State: Running, Role: LEADER
I20260928 03:04:50.405819 12358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d745fbe3b1460bb77b6b88fa014e96" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43331 } }
I20260928 03:04:50.406442 12353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:04:50.408834 12359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73d745fbe3b1460bb77b6b88fa014e96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d745fbe3b1460bb77b6b88fa014e96" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43331 } } }
I20260928 03:04:50.409353 12359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:50.408998 12360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73d745fbe3b1460bb77b6b88fa014e96. Latest consensus state: current_term: 1 leader_uuid: "73d745fbe3b1460bb77b6b88fa014e96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d745fbe3b1460bb77b6b88fa014e96" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43331 } } }
I20260928 03:04:50.409778 12360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:50.411100 12367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:04:50.416673 12367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:04:50.426685 12367 catalog_manager.cc:1347] Generated new cluster ID: d851f968b28546a78f2d2516d5179409
I20260928 03:04:50.426838 12367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:04:50.443178 12367 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:04:50.445184 12367 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:04:50.457841 12367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73d745fbe3b1460bb77b6b88fa014e96: Generated new TSK 0
I20260928 03:04:50.458817 12367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:04:50.489609 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:50.480598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50808 (local address 127.6.192.62:43331)
0928 03:04:50.480849 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:50.480857 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:50.481192 (+ 335us) server_negotiation.cc:407] Connection header received
0928 03:04:50.481410 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:50.481416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:50.481524 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:50.481711 (+ 187us) server_negotiation.cc:226] Negotiated authn=JWT
0928 03:04:50.482145 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:50.483298 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:50.484808 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:50.485544 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:50.485686 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:50.486411 (+ 725us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 03:04:50.488916 (+ 2505us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46459/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 03:04:50.489280 (+ 364us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50808: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46459/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":82}
W20260928 03:04:50.489748 12356 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50808: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46459/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 03:04:50.491997 12382 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:04:50.480434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43331 (local address 127.0.0.1:50808)
0928 03:04:50.481080 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:50.481096 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:50.481240 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:50.481727 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:50.481731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:50.481742 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0928 03:04:50.481991 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:50.481999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:50.483517 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:50.483521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:50.484644 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:50.484658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:50.485817 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:50.485823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:50.486056 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:50.486100 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 03:04:50.489162 (+ 3062us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46459/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 03:04:50.491524 (+ 2362us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.192.62:43331: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46459/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":1855,"thread_start_us":341,"threads_started":1}
I20260928 03:04:50.494006 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 12287
I20260928 03:04:50.495484 12348 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 03:04:50.503053 12383 webserver.cc:533] Webserver: sslize failed:
E20260928 03:04:50.503201 12383 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 03:04:50.588126 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 12287
2026-09-28T03:04:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (762 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T03:04:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:04:50Z Disabled control of system clock
I20260928 03:04:50.673228 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:40595
--webserver_interface=127.6.192.62
--webserver_port=0
--builtin_ntp_servers=127.6.192.20:46651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:40595
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564644636890-6912-0/testchainca.cert with env {}
W20260928 03:04:51.023845 12388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:51.024165 12388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:51.024333 12388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:51.032003 12388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:04:51.032128 12388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:51.032198 12388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:51.032294 12388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:04:51.032382 12388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 03:04:51.032477 12388 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 03:04:51.032531 12388 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 03:04:51.042887 12388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:40595
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:40595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564644636890-6912-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:51.044374 12388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:51.045802 12388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:51.060735 12394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:51.061007 12393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:51.061048 12396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:51.061976 12388 server_base.cc:1034] running on GCE node
I20260928 03:04:51.062860 12388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:51.064518 12388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:51.065781 12388 hybrid_clock.cc:648] HybridClock initialized: now 1790564691065679 us; error 61 us; skew 500 ppm
I20260928 03:04:51.066123 12388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:04:51.068965 12388 webserver.cc:460] Webserver started at http://127.6.192.62:37721/ using document root <none> and password file <none>
I20260928 03:04:51.069676 12388 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:51.069782 12388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:51.070058 12388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:51.073205 12388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "c2fa120afd04470294258102971cbca9"
format_stamp: "Formatted at 2026-09-28 03:04:51 on dist-test-slave-fjww"
server_key: "557f4d23bc69d30ec4b89ab194d16772"
server_key_iv: "7bf4bb93505d1e58d08312320ab6ec93"
server_key_version: "encryptionkey@0"
I20260928 03:04:51.074038 12388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "c2fa120afd04470294258102971cbca9"
format_stamp: "Formatted at 2026-09-28 03:04:51 on dist-test-slave-fjww"
server_key: "557f4d23bc69d30ec4b89ab194d16772"
server_key_iv: "7bf4bb93505d1e58d08312320ab6ec93"
server_key_version: "encryptionkey@0"
I20260928 03:04:51.079999 12388 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 03:04:51.084163 12402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:51.085897 12388 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 03:04:51.086123 12388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "c2fa120afd04470294258102971cbca9"
format_stamp: "Formatted at 2026-09-28 03:04:51 on dist-test-slave-fjww"
server_key: "557f4d23bc69d30ec4b89ab194d16772"
server_key_iv: "7bf4bb93505d1e58d08312320ab6ec93"
server_key_version: "encryptionkey@0"
I20260928 03:04:51.086426 12388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:51.105525 12388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:51.106473 12388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:51.106803 12388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:51.127142 12388 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:40595
I20260928 03:04:51.127148 12453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:40595 every 8 connection(s)
I20260928 03:04:51.128731 12388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:04:51.129228 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 12388
I20260928 03:04:51.129457 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:04:51.130383 6912 external_mini_cluster.cc:1428] Setting key 7f5567099643f924ee92b09bbefb4d58
I20260928 03:04:51.134110 12454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:51.145077 12457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:51.132592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34672 (local address 127.6.192.62:40595)
0928 03:04:51.133664 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:51.133676 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:51.133719 (+ 43us) server_negotiation.cc:407] Connection header received
0928 03:04:51.135586 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:51.135614 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:51.136113 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:51.136760 (+ 647us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:51.137387 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:51.138694 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:51.140130 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:51.140907 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:51.141133 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:51.142443 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:51.142452 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:51.142469 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:04:51.143049 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:51.143136 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:04:51.143158 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:51.143322 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:51.143570 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 03:04:51.143936 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":372,"threads_started":1}
I20260928 03:04:51.147158 12454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9: Bootstrap starting.
I20260928 03:04:51.149093 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:04:51.150856 12454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:51.152246 12454 log.cc:824] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9: Log is configured to *not* fsync() on all Append() calls
I20260928 03:04:51.155800 12454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9: No bootstrap required, opened a new log
I20260928 03:04:51.165110 12454 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2fa120afd04470294258102971cbca9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40595 } }
I20260928 03:04:51.165599 12454 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:51.165728 12454 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2fa120afd04470294258102971cbca9, State: Initialized, Role: FOLLOWER
I20260928 03:04:51.166391 12454 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2fa120afd04470294258102971cbca9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40595 } }
I20260928 03:04:51.166677 12454 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:04:51.166837 12454 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:04:51.167030 12454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:51.169832 12454 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2fa120afd04470294258102971cbca9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40595 } }
I20260928 03:04:51.170517 12454 leader_election.cc:302] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2fa120afd04470294258102971cbca9; no voters:
I20260928 03:04:51.171198 12454 leader_election.cc:288] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:04:51.171370 12459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:51.172250 12459 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9 [term 1 LEADER]: Becoming Leader. State: Replica: c2fa120afd04470294258102971cbca9, State: Running, Role: LEADER
I20260928 03:04:51.172832 12459 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2fa120afd04470294258102971cbca9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40595 } }
I20260928 03:04:51.173599 12454 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:04:51.175962 12461 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2fa120afd04470294258102971cbca9. Latest consensus state: current_term: 1 leader_uuid: "c2fa120afd04470294258102971cbca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2fa120afd04470294258102971cbca9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40595 } } }
I20260928 03:04:51.176489 12461 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:51.177068 12460 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2fa120afd04470294258102971cbca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2fa120afd04470294258102971cbca9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40595 } } }
I20260928 03:04:51.177455 12460 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:51.180007 12466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:04:51.185907 12466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:04:51.197281 12466 catalog_manager.cc:1347] Generated new cluster ID: b760401b13cc4c33a923d3f0392adc80
I20260928 03:04:51.197424 12466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:04:51.217988 12466 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:04:51.219847 12466 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:04:51.232225 12466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2fa120afd04470294258102971cbca9: Generated new TSK 0
I20260928 03:04:51.233485 12466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:04:51.306640 12457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:51.261971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36600 (local address 127.6.192.62:40595)
0928 03:04:51.262249 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:51.262259 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:51.262817 (+ 558us) server_negotiation.cc:407] Connection header received
0928 03:04:51.263137 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:51.263141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:51.263318 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:51.263594 (+ 276us) server_negotiation.cc:226] Negotiated authn=JWT
0928 03:04:51.264354 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:51.265159 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:51.267593 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:51.268078 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:51.268188 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:51.268910 (+ 722us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 03:04:51.306026 (+ 37116us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 03:04:51.306386 (+ 360us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36600: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":82}
W20260928 03:04:51.306973 12457 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36600: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 03:04:51.307143 12483 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:04:51.261902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:40595 (local address 127.0.0.1:36600)
0928 03:04:51.262668 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:51.262692 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:51.262894 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:51.263632 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:51.263639 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:51.263656 (+ 17us) client_negotiation.cc:189] Negotiated authn=JWT
0928 03:04:51.264081 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:51.264095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:51.265350 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:51.265357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:51.267326 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:51.267348 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:51.268295 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:51.268303 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:51.268553 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:51.268610 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 03:04:51.306378 (+ 37768us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 03:04:51.306722 (+ 344us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.192.62:40595: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":393,"threads_started":1}
I20260928 03:04:51.309080 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 12388
I20260928 03:04:51.310345 12449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:51.410835 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 12388
2026-09-28T03:04:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (801 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (448 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (436 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:04:52 dist-test-slave-fjww krb5kdc[12498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:04:52 dist-test-slave-fjww krb5kdc[12498](info): set up 2 sockets
Sep 28 03:04:52 dist-test-slave-fjww krb5kdc[12498](info): commencing operation
krb5kdc: starting...
W20260928 03:04:53.420485 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:04:53 dist-test-slave-fjww krb5kdc[12498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:04:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:04:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:04:53.644114 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:44609
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:44609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:53.979569 12511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:53.979950 12511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:53.980125 12511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:53.987533 12511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:04:53.987703 12511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:53.987815 12511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:53.987918 12511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:04:53.988006 12511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:04:53.998239 12511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:44609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:44609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:53.999732 12511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:54.001075 12511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:54.013463 12519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:54.013525 12516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:54.013475 12517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:54.014942 12511 server_base.cc:1034] running on GCE node
I20260928 03:04:54.016193 12511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:54.017987 12511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:54.019315 12511 hybrid_clock.cc:648] HybridClock initialized: now 1790564694019227 us; error 68 us; skew 500 ppm
I20260928 03:04:54.019872 12511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:54 dist-test-slave-fjww krb5kdc[12498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564694, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:54.023890 12511 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:04:54.025672 12511 webserver.cc:460] Webserver started at http://127.6.192.62:40897/ using document root <none> and password file <none>
I20260928 03:04:54.026470 12511 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:54.026599 12511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:54.026927 12511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:54.030571 12511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "0050d89499e4431498161b7c99b00891"
format_stamp: "Formatted at 2026-09-28 03:04:54 on dist-test-slave-fjww"
server_key: "404391f76836aed8bbe5a28c5311312d"
server_key_iv: "f7720c4397b6276592a801264fb348e6"
server_key_version: "encryptionkey@0"
I20260928 03:04:54.031394 12511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "0050d89499e4431498161b7c99b00891"
format_stamp: "Formatted at 2026-09-28 03:04:54 on dist-test-slave-fjww"
server_key: "404391f76836aed8bbe5a28c5311312d"
server_key_iv: "f7720c4397b6276592a801264fb348e6"
server_key_version: "encryptionkey@0"
I20260928 03:04:54.037242 12511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 03:04:54.040967 12526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:54.042511 12511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:04:54.042742 12511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "0050d89499e4431498161b7c99b00891"
format_stamp: "Formatted at 2026-09-28 03:04:54 on dist-test-slave-fjww"
server_key: "404391f76836aed8bbe5a28c5311312d"
server_key_iv: "f7720c4397b6276592a801264fb348e6"
server_key_version: "encryptionkey@0"
I20260928 03:04:54.042961 12511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:54.063616 12511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:54.068713 12511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:54.069032 12511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:54.089001 12511 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:44609
I20260928 03:04:54.089037 12577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:44609 every 8 connection(s)
I20260928 03:04:54.090428 12511 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:04:54.094394 12578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:54.096771 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 12511
I20260928 03:04:54.097028 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:04:54.097724 6912 external_mini_cluster.cc:1428] Setting key 6a69bbdd421c84f291cf88a6793b1b07
I20260928 03:04:54.107064 12578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891: Bootstrap starting.
I20260928 03:04:54.110816 12578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:54.112241 12578 log.cc:824] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891: Log is configured to *not* fsync() on all Append() calls
Sep 28 03:04:54 dist-test-slave-fjww krb5kdc[12498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:54.116514 12578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891: No bootstrap required, opened a new log
I20260928 03:04:54.124069 12581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:54.099854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.6.192.62:44609)
0928 03:04:54.100672 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:54.100686 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:54.100730 (+ 44us) server_negotiation.cc:407] Connection header received
0928 03:04:54.102575 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:54.102600 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:54.103049 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:54.103624 (+ 575us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:54.105138 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:54.106467 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:54.107881 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:54.108585 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:54.108805 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:54.113908 (+ 5103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:54.113924 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:54.113950 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:54.113998 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:54.118409 (+ 4411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:54.119332 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:54.119343 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:54.119353 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:54.119669 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:54.120156 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:54.120162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:54.120166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:54.121029 (+ 863us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:54.121364 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:54.121874 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 03:04:54.122389 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":354,"threads_started":1}
I20260928 03:04:54.128283 12578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0050d89499e4431498161b7c99b00891" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 44609 } }
I20260928 03:04:54.128671 12578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:54.128763 12578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0050d89499e4431498161b7c99b00891, State: Initialized, Role: FOLLOWER
I20260928 03:04:54.129333 12578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0050d89499e4431498161b7c99b00891" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 44609 } }
I20260928 03:04:54.129552 12578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:04:54.129660 12578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:04:54.129808 12578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:54.132606 12578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0050d89499e4431498161b7c99b00891" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 44609 } }
I20260928 03:04:54.133112 12578 leader_election.cc:302] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0050d89499e4431498161b7c99b00891; no voters:
I20260928 03:04:54.133709 12578 leader_election.cc:288] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:04:54.133893 12583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:54.134627 12583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891 [term 1 LEADER]: Becoming Leader. State: Replica: 0050d89499e4431498161b7c99b00891, State: Running, Role: LEADER
I20260928 03:04:54.135267 12583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0050d89499e4431498161b7c99b00891" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 44609 } }
I20260928 03:04:54.136279 12578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:04:54.138382 12585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0050d89499e4431498161b7c99b00891. Latest consensus state: current_term: 1 leader_uuid: "0050d89499e4431498161b7c99b00891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0050d89499e4431498161b7c99b00891" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 44609 } } }
I20260928 03:04:54.138617 12584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0050d89499e4431498161b7c99b00891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0050d89499e4431498161b7c99b00891" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 44609 } } }
I20260928 03:04:54.139087 12585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:54.139087 12584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:54.140432 12591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:04:54.147403 12591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:04:54.157332 12591 catalog_manager.cc:1347] Generated new cluster ID: a4595af04f0d42e3afed9ef51d358fbe
I20260928 03:04:54.157527 12591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:04:54.170197 12591 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:04:54.171716 12591 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:04:54.192481 12591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0050d89499e4431498161b7c99b00891: Generated new TSK 0
I20260928 03:04:54.193670 12591 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
I20260928 03:04:54.259873 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:44609
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:54.603643 12604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:54.604000 12604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:54.604180 12604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:54.612337 12604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:54.612500 12604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:54.612671 12604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:04:54.622823 12604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:44609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:54.624301 12604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:54.625679 12604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:54.639911 12610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:54.639938 12612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:54.640067 12609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:54.640966 12604 server_base.cc:1034] running on GCE node
I20260928 03:04:54.641660 12604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:54.642664 12604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:54.643891 12604 hybrid_clock.cc:648] HybridClock initialized: now 1790564694643829 us; error 68 us; skew 500 ppm
I20260928 03:04:54.644193 12604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:54 dist-test-slave-fjww krb5kdc[12498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564694, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:54.647908 12604 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:04:54.649633 12604 webserver.cc:460] Webserver started at http://127.6.192.1:44469/ using document root <none> and password file <none>
I20260928 03:04:54.650413 12604 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:54.650552 12604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:54.650916 12604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:54.654034 12604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "0f5bf0d0918c408094cb0ca5d545d9b0"
format_stamp: "Formatted at 2026-09-28 03:04:54 on dist-test-slave-fjww"
server_key: "441975a0c3d8e99f1c6b89f5083726ca"
server_key_iv: "c1b4786ff6cf3f0b5f796dc34a90cb4f"
server_key_version: "encryptionkey@0"
I20260928 03:04:54.654896 12604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "0f5bf0d0918c408094cb0ca5d545d9b0"
format_stamp: "Formatted at 2026-09-28 03:04:54 on dist-test-slave-fjww"
server_key: "441975a0c3d8e99f1c6b89f5083726ca"
server_key_iv: "c1b4786ff6cf3f0b5f796dc34a90cb4f"
server_key_version: "encryptionkey@0"
I20260928 03:04:54.660804 12604 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260928 03:04:54.664940 12619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:54.666721 12604 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260928 03:04:54.666931 12604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "0f5bf0d0918c408094cb0ca5d545d9b0"
format_stamp: "Formatted at 2026-09-28 03:04:54 on dist-test-slave-fjww"
server_key: "441975a0c3d8e99f1c6b89f5083726ca"
server_key_iv: "c1b4786ff6cf3f0b5f796dc34a90cb4f"
server_key_version: "encryptionkey@0"
I20260928 03:04:54.667151 12604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:54.684911 12604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:54.690311 12604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:54.690658 12604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:54.691861 12604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:54.693531 12604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:54.693619 12604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:54.693763 12604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:54.693832 12604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:54.732064 12604 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:34179
I20260928 03:04:54.732102 12731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:34179 every 8 connection(s)
I20260928 03:04:54.733541 12604 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:04:54.734253 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 12604
I20260928 03:04:54.734485 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:04:54.735376 6912 external_mini_cluster.cc:1428] Setting key 6e335f8ae9f2c3b53641a3df221d0ce0
Sep 28 03:04:54 dist-test-slave-fjww krb5kdc[12498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564694, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.192.2@KRBTEST.COM; defaulting to no policy
I20260928 03:04:54.757663 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:54.736683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:45919 (local address 127.6.192.62:44609)
0928 03:04:54.737169 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:54.737175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:54.738560 (+ 1385us) server_negotiation.cc:407] Connection header received
0928 03:04:54.740687 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:54.740696 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:54.740799 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:54.740929 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:54.742554 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:54.743543 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:54.746262 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:54.746784 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:54.746897 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:54.751498 (+ 4601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:54.751513 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:54.751517 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:54.751554 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:54.754250 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:54.755065 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:54.755070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:54.755074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:54.755209 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:54.755747 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:54.755753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:54.755757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:54.756041 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:54.756213 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:54.757015 (+ 802us) server_negotiation.cc:299] Negotiation successful
0928 03:04:54.757250 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":246,"threads_started":1}
I20260928 03:04:54.758719 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:54.737429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:44609 (local address 127.6.192.1:45919)
0928 03:04:54.738373 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:54.738423 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:54.740393 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:54.741177 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:54.741191 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:54.741766 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:54.742364 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:54.742383 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:54.743716 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:54.743721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:54.745927 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:54.745945 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:54.747052 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:54.747061 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:54.747430 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:54.748252 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:54.748287 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:54.751112 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:54.754435 (+ 3323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:54.754446 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:54.754467 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:54.754930 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:54.755323 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:54.755329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:54.755333 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:54.755624 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:54.756192 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:54.756202 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:54.756632 (+ 430us) client_negotiation.cc:769] Sending connection context
0928 03:04:54.756919 (+ 287us) client_negotiation.cc:240] Negotiation successful
0928 03:04:54.757491 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":403,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.2@KRBTEST.COM" created.
I20260928 03:04:54.760500 12732 heartbeater.cc:347] Connected to a master server at 127.6.192.62:44609
I20260928 03:04:54.760829 12732 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:54.761610 12732 heartbeater.cc:511] Master 127.6.192.62:44609 requested a full tablet report, sending...
I20260928 03:04:54.763971 12543 ts_manager.cc:194] Registered new tserver with Master: 0f5bf0d0918c408094cb0ca5d545d9b0 (127.6.192.1:34179)
I20260928 03:04:54.765457 12543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:45919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
I20260928 03:04:54.787699 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:44609
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:55.165597 12738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:55.165966 12738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:55.166078 12738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:55.174636 12738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:55.174768 12738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:55.174894 12738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:04:55.185878 12738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:44609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:55.187183 12738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:55.188674 12738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:55.203050 12743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:55.203013 12746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:55.202971 12744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:55.204000 12738 server_base.cc:1034] running on GCE node
I20260928 03:04:55.204747 12738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:55.205794 12738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:55.207046 12738 hybrid_clock.cc:648] HybridClock initialized: now 1790564695206965 us; error 54 us; skew 500 ppm
I20260928 03:04:55.207494 12738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:55 dist-test-slave-fjww krb5kdc[12498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564695, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:55.211828 12738 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:04:55.213459 12738 webserver.cc:460] Webserver started at http://127.6.192.2:34115/ using document root <none> and password file <none>
I20260928 03:04:55.214182 12738 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:55.214318 12738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:55.214586 12738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:55.217803 12738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "b872af2b6db34fac982e462b76cc38f5"
format_stamp: "Formatted at 2026-09-28 03:04:55 on dist-test-slave-fjww"
server_key: "3614c29337f942d44ed734d40d867bd2"
server_key_iv: "a8fae01b13e22e8d9a4548079eda4980"
server_key_version: "encryptionkey@0"
I20260928 03:04:55.218598 12738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "b872af2b6db34fac982e462b76cc38f5"
format_stamp: "Formatted at 2026-09-28 03:04:55 on dist-test-slave-fjww"
server_key: "3614c29337f942d44ed734d40d867bd2"
server_key_iv: "a8fae01b13e22e8d9a4548079eda4980"
server_key_version: "encryptionkey@0"
I20260928 03:04:55.224613 12738 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260928 03:04:55.228968 12753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:55.230751 12738 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 03:04:55.230989 12738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "b872af2b6db34fac982e462b76cc38f5"
format_stamp: "Formatted at 2026-09-28 03:04:55 on dist-test-slave-fjww"
server_key: "3614c29337f942d44ed734d40d867bd2"
server_key_iv: "a8fae01b13e22e8d9a4548079eda4980"
server_key_version: "encryptionkey@0"
I20260928 03:04:55.231240 12738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:55.252318 12738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:55.257123 12738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:55.257459 12738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:55.258610 12738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:55.260365 12738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:55.260478 12738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:55.260644 12738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:55.260731 12738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:55.298622 12738 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:45443
I20260928 03:04:55.298645 12865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:45443 every 8 connection(s)
I20260928 03:04:55.300132 12738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:04:55.304840 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 12738
I20260928 03:04:55.305029 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:04:55.305616 6912 external_mini_cluster.cc:1428] Setting key 1c3ee8b91dd368fe64fd1efe27ac51f8
Sep 28 03:04:55 dist-test-slave-fjww krb5kdc[12498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564695, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.192.3@KRBTEST.COM; defaulting to no policy
I20260928 03:04:55.329341 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:55.302845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:50327 (local address 127.6.192.62:44609)
0928 03:04:55.303312 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:55.303317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:55.305059 (+ 1742us) server_negotiation.cc:407] Connection header received
0928 03:04:55.306875 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:55.306879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:55.306957 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:55.307056 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:55.309123 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:55.310001 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:55.312627 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:55.313136 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:55.313345 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:55.321938 (+ 8593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:55.321944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:55.321948 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:55.321991 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:55.325362 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:55.326355 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:55.326360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:55.326363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:55.326518 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:55.327074 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:55.327085 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:55.327088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:55.327422 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:55.327683 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:55.328607 (+ 924us) server_negotiation.cc:299] Negotiation successful
0928 03:04:55.328874 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":244,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.3@KRBTEST.COM" created.
I20260928 03:04:55.330749 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:55.303360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:44609 (local address 127.6.192.2:50327)
0928 03:04:55.304199 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:55.304238 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:55.306575 (+ 2337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:55.307366 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:55.307378 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:55.308044 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:55.308910 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:55.308939 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:55.310188 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:55.310197 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:55.312443 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:55.312463 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:55.313475 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:55.313485 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:55.313822 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:55.318214 (+ 4392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:55.318261 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:55.321553 (+ 3292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:55.325580 (+ 4027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:55.325592 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:55.325615 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:55.326168 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:55.326669 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:55.326674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:55.326679 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:55.326920 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:55.327637 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:55.327648 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:55.328150 (+ 502us) client_negotiation.cc:769] Sending connection context
0928 03:04:55.328503 (+ 353us) client_negotiation.cc:240] Negotiation successful
0928 03:04:55.329228 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":339,"threads_started":1}
I20260928 03:04:55.332636 12866 heartbeater.cc:347] Connected to a master server at 127.6.192.62:44609
I20260928 03:04:55.333009 12866 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:55.333988 12866 heartbeater.cc:511] Master 127.6.192.62:44609 requested a full tablet report, sending...
I20260928 03:04:55.335918 12543 ts_manager.cc:194] Registered new tserver with Master: b872af2b6db34fac982e462b76cc38f5 (127.6.192.2:45443)
I20260928 03:04:55.336941 12543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:50327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
I20260928 03:04:55.359891 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:44609
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:55.710305 12872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:55.710625 12872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:55.710881 12872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:55.719012 12872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:55.719153 12872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:55.719352 12872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:04:55.729660 12872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:44609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:55.731127 12872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:55.732566 12872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:55.747542 12880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:55.747659 12878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:55.747778 12877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:55.748728 12872 server_base.cc:1034] running on GCE node
I20260928 03:04:55.749390 12872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:55.750345 12872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:55.751552 12872 hybrid_clock.cc:648] HybridClock initialized: now 1790564695751483 us; error 55 us; skew 500 ppm
I20260928 03:04:55.751909 12872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:55 dist-test-slave-fjww krb5kdc[12498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564695, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:55.755191 12872 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:04:55.756850 12872 webserver.cc:460] Webserver started at http://127.6.192.3:45461/ using document root <none> and password file <none>
I20260928 03:04:55.757612 12872 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:55.757750 12872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:55.758080 12872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:55.761133 12872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "d383ce4ccd2341c88834ac7801792aa1"
format_stamp: "Formatted at 2026-09-28 03:04:55 on dist-test-slave-fjww"
server_key: "193997c5f48b8febc078a03f91e09bf4"
server_key_iv: "40dcffb548a4ebaca394f4e62a28e35f"
server_key_version: "encryptionkey@0"
I20260928 03:04:55.761950 12872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "d383ce4ccd2341c88834ac7801792aa1"
format_stamp: "Formatted at 2026-09-28 03:04:55 on dist-test-slave-fjww"
server_key: "193997c5f48b8febc078a03f91e09bf4"
server_key_iv: "40dcffb548a4ebaca394f4e62a28e35f"
server_key_version: "encryptionkey@0"
I20260928 03:04:55.767319 12872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 03:04:55.768903 12732 heartbeater.cc:503] Master 127.6.192.62:44609 was elected leader, sending a full tablet report...
I20260928 03:04:55.771587 12887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:55.773088 12872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:04:55.773229 12872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "d383ce4ccd2341c88834ac7801792aa1"
format_stamp: "Formatted at 2026-09-28 03:04:55 on dist-test-slave-fjww"
server_key: "193997c5f48b8febc078a03f91e09bf4"
server_key_iv: "40dcffb548a4ebaca394f4e62a28e35f"
server_key_version: "encryptionkey@0"
I20260928 03:04:55.773419 12872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:55.802917 12872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:55.807542 12872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:55.807801 12872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:55.808776 12872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:55.810170 12872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:55.810251 12872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:55.810338 12872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:55.810384 12872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:55.847219 12872 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:41385
I20260928 03:04:55.847352 12999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:41385 every 8 connection(s)
I20260928 03:04:55.848681 12872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:04:55.852715 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 12872
I20260928 03:04:55.852926 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:04:55.853385 6912 external_mini_cluster.cc:1428] Setting key 3313bdefdea1a5c1ea528a15bbcab1de
Sep 28 03:04:55 dist-test-slave-fjww krb5kdc[12498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564695, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:04:55.870860 13002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:55.851683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:48583 (local address 127.6.192.62:44609)
0928 03:04:55.852130 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:55.852135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:55.853541 (+ 1406us) server_negotiation.cc:407] Connection header received
0928 03:04:55.855582 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:55.855587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:55.855687 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:55.855807 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:55.857530 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:55.858389 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:55.860066 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:55.860465 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:55.860568 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:55.864224 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:55.864229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:55.864233 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:55.864268 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:55.867157 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:55.868172 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:55.868177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:55.868190 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:55.868314 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:55.869016 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:55.869020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:55.869022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:55.869302 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:55.869460 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:55.870211 (+ 751us) server_negotiation.cc:299] Negotiation successful
0928 03:04:55.870439 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":213,"threads_started":1}
I20260928 03:04:55.872025 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:55.852345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:44609 (local address 127.6.192.3:48583)
0928 03:04:55.853169 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:55.853223 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:55.855208 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:55.856063 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:55.856074 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:55.856744 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:55.857362 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:55.857382 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:55.858532 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:55.858537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:55.859925 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:55.859937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:55.860664 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:55.860669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:55.860881 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:55.861536 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:55.861559 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:55.863943 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:55.867347 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:55.867358 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:55.867384 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:55.868041 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:55.868429 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:55.868434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:55.868438 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:55.868643 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:55.869435 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:55.869443 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:55.869821 (+ 378us) client_negotiation.cc:769] Sending connection context
0928 03:04:55.870088 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 03:04:55.870693 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":357,"threads_started":1}
I20260928 03:04:55.873454 13000 heartbeater.cc:347] Connected to a master server at 127.6.192.62:44609
I20260928 03:04:55.873739 13000 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:55.874418 13000 heartbeater.cc:511] Master 127.6.192.62:44609 requested a full tablet report, sending...
I20260928 03:04:55.875916 12543 ts_manager.cc:194] Registered new tserver with Master: d383ce4ccd2341c88834ac7801792aa1 (127.6.192.3:41385)
I20260928 03:04:55.876972 12543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:48583
I20260928 03:04:55.881973 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/test-admin.keytab.
Sep 28 03:04:55 dist-test-slave-fjww krb5kdc[12498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:55.909819 6912 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 03:04:55.915299 13008 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:04:55.912722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49452 (local address 127.6.192.1:34179)
0928 03:04:55.913403 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:55.913410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:55.913438 (+ 28us) server_negotiation.cc:407] Connection header received
0928 03:04:55.913498 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:55.913504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:55.913714 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:55.913843 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:55.914860 (+ 1017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49452: BlockingRecv error: recv got EOF from 127.0.0.1:49452 (error 108)
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":333,"threads_started":1}
W20260928 03:04:55.915473 6912 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/krb5cc' of type FILE
Sep 28 03:04:55 dist-test-slave-fjww krb5kdc[12498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:55.917117 6912 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 03:04:55 dist-test-slave-fjww krb5kdc[12498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:55.918911 6912 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 03:04:55 dist-test-slave-fjww krb5kdc[12498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:04:55.938884 13008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:55.920901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49460 (local address 127.6.192.1:34179)
0928 03:04:55.921176 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:55.921192 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:55.921447 (+ 255us) server_negotiation.cc:407] Connection header received
0928 03:04:55.921724 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:55.921731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:55.921825 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:55.921955 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:55.923516 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:55.924660 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:55.926722 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:55.927650 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:55.927832 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:55.932364 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:55.932378 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:55.932394 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:55.932435 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:55.935754 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:55.936654 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:55.936664 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:55.936666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:55.936799 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:55.937255 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:55.937262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:55.937265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:55.937650 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:55.937955 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:55.938324 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 03:04:55.938591 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 03:04:55.945551 13008 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:04:55.943186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.6.192.1:34179)
0928 03:04:55.943616 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:55.943625 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:55.943920 (+ 295us) server_negotiation.cc:407] Connection header received
0928 03:04:55.944149 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:55.944153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:55.944237 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:55.944351 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:55.945274 (+ 923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49476: BlockingRecv error: recv got EOF from 127.0.0.1:49476 (error 108)
Metrics: {"server-negotiator.queue_time_us":228}
W20260928 03:04:55.945894 6912 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/krb5cc' of type FILE
Sep 28 03:04:55 dist-test-slave-fjww krb5kdc[12498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:55.947953 6912 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 03:04:55 dist-test-slave-fjww krb5kdc[12498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:55.949509 6912 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 03:04:55 dist-test-slave-fjww krb5kdc[12498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:04:55.965175 13008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:55.951015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49486 (local address 127.6.192.1:34179)
0928 03:04:55.951243 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:55.951249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:55.951603 (+ 354us) server_negotiation.cc:407] Connection header received
0928 03:04:55.951799 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:55.951811 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:55.951872 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:55.951976 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:55.952926 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:55.953627 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:55.954855 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:55.955289 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:55.955441 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:55.959257 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:55.959265 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:55.959269 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:55.959321 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:55.961964 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:55.962899 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:55.962908 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:55.962917 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:55.963061 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:55.963636 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:55.963648 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:55.963652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:55.964006 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:55.964313 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:55.964631 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 03:04:55.964867 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260928 03:04:55.970168 13008 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:04:55.968266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49492 (local address 127.6.192.1:34179)
0928 03:04:55.968502 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:55.968508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:55.968776 (+ 268us) server_negotiation.cc:407] Connection header received
0928 03:04:55.968970 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:55.968975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:55.969055 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:55.969157 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:55.969927 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49492: BlockingRecv error: recv got EOF from 127.0.0.1:49492 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260928 03:04:55.970299 6912 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790564644636890-6912-0/krb5kdc/krb5cc' of type FILE
Sep 28 03:04:55 dist-test-slave-fjww krb5kdc[12498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:55.972117 6912 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 03:04:55 dist-test-slave-fjww krb5kdc[12498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:55.973676 6912 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 03:04:55 dist-test-slave-fjww krb5kdc[12498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:04:55.989972 13008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:55.975130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.6.192.1:34179)
0928 03:04:55.975369 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:55.975376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:55.975675 (+ 299us) server_negotiation.cc:407] Connection header received
0928 03:04:55.975849 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:55.975853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:55.975917 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:55.976000 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:55.977117 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:55.977960 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:55.979387 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:55.979857 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:55.979953 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:55.983953 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:55.983958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:55.983961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:55.984000 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:55.987080 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:55.987933 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:55.987937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:55.987939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:55.988096 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:55.988566 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:55.988569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:55.988571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:55.988838 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:55.989048 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:55.989389 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 03:04:55.989563 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 03:04:55.992076 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 12604
I20260928 03:04:56.005272 13008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:55.992962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49510 (local address 127.6.192.1:34179)
0928 03:04:55.993147 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:55.993151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:55.993464 (+ 313us) server_negotiation.cc:407] Connection header received
0928 03:04:55.993576 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:55.993599 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:55.993664 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:55.993762 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:55.994746 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:55.995719 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:55.996957 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:55.997460 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:55.997562 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:55.999910 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:55.999915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:55.999918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:55.999951 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:56.002682 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:56.003355 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:56.003359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:56.003362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:56.003602 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:56.003985 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:56.003989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:56.003991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:56.004279 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:56.004512 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:56.004803 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 03:04:56.004990 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 03:04:56.005903 12727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:56.091572 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 12604
I20260928 03:04:56.108461 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 12738
Sep 28 03:04:56 dist-test-slave-fjww krb5kdc[12498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.2@KRBTEST.COM
I20260928 03:04:56.126462 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:56.109574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.6.192.2:45443)
0928 03:04:56.110150 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:56.110157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:56.110182 (+ 25us) server_negotiation.cc:407] Connection header received
0928 03:04:56.110255 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:56.110261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:56.110491 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:56.110628 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:56.111806 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:56.112867 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:56.114190 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:56.114794 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:56.114909 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:56.119243 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:56.119259 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:56.119276 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:56.119373 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:56.122630 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:56.123893 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:56.123902 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:56.123905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:56.124085 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:56.124595 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:56.124599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:56.124602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:56.125035 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:56.125389 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:56.125739 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 03:04:56.126111 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":300,"threads_started":1}
I20260928 03:04:56.127936 12861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:56.214653 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 12738
I20260928 03:04:56.231632 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 12872
Sep 28 03:04:56 dist-test-slave-fjww krb5kdc[12498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.3@KRBTEST.COM
I20260928 03:04:56.249141 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:56.232854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57458 (local address 127.6.192.3:41385)
0928 03:04:56.233537 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:56.233544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:56.233575 (+ 31us) server_negotiation.cc:407] Connection header received
0928 03:04:56.233635 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:56.233673 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:56.233970 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:56.234110 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:56.235287 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:56.236464 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:56.237956 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:56.238573 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:56.238710 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:56.242794 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:56.242809 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:56.242825 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:56.242867 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:56.246266 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:56.247042 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:56.247051 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:56.247054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:56.247221 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:56.247664 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:56.247668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:56.247670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:56.248081 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:56.248435 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:56.248628 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 03:04:56.248849 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":314,"threads_started":1}
I20260928 03:04:56.250512 12995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:56.337299 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 12872
I20260928 03:04:56.354072 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 12511
I20260928 03:04:56.355609 12573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:04:56.462476 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 12511
2026-09-28T03:04:56Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4172 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:04:56 dist-test-slave-fjww krb5kdc[13028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:04:56 dist-test-slave-fjww krb5kdc[13028](info): set up 2 sockets
Sep 28 03:04:56 dist-test-slave-fjww krb5kdc[13028](info): commencing operation
krb5kdc: starting...
W20260928 03:04:57.592435 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:04:57 dist-test-slave-fjww krb5kdc[13028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:04:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:04:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.62.keytab.
Entry for principal oryx/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.62.keytab.
I20260928 03:04:57.823982 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:41507
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.62.keytab
--principal=oryx/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:41507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:58.158795 13041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:58.159112 13041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:58.159277 13041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:58.166837 13041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:04:58.166958 13041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:58.167053 13041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:58.167155 13041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:04:58.167219 13041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:04:58.177419 13041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:41507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.62.keytab
--principal=oryx/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:41507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:58.178844 13041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:58.180207 13041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:58.192580 13047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:58.192649 13049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:58.192620 13046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:58.193444 13041 server_base.cc:1034] running on GCE node
I20260928 03:04:58.194231 13041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:58.195690 13041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:58.196904 13041 hybrid_clock.cc:648] HybridClock initialized: now 1790564698196853 us; error 38 us; skew 500 ppm
I20260928 03:04:58.197268 13041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:58 dist-test-slave-fjww krb5kdc[13028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564698, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:58.200870 13041 init.cc:377] Logged in from keytab as oryx/127.6.192.62@KRBTEST.COM (short username oryx)
I20260928 03:04:58.202710 13041 webserver.cc:460] Webserver started at http://127.6.192.62:33127/ using document root <none> and password file <none>
I20260928 03:04:58.203547 13041 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:58.203708 13041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:58.204046 13041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:58.207494 13041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "7a2716d3b7d545dd82f8fadedefd92d9"
format_stamp: "Formatted at 2026-09-28 03:04:58 on dist-test-slave-fjww"
server_key: "9529276976eaf191c42f2ee46654b2c6"
server_key_iv: "39487253dcaa3e98f1b377e08ff19c48"
server_key_version: "encryptionkey@0"
I20260928 03:04:58.208369 13041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "7a2716d3b7d545dd82f8fadedefd92d9"
format_stamp: "Formatted at 2026-09-28 03:04:58 on dist-test-slave-fjww"
server_key: "9529276976eaf191c42f2ee46654b2c6"
server_key_iv: "39487253dcaa3e98f1b377e08ff19c48"
server_key_version: "encryptionkey@0"
I20260928 03:04:58.214360 13041 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260928 03:04:58.218235 13056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:58.220278 13041 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:04:58.220489 13041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "7a2716d3b7d545dd82f8fadedefd92d9"
format_stamp: "Formatted at 2026-09-28 03:04:58 on dist-test-slave-fjww"
server_key: "9529276976eaf191c42f2ee46654b2c6"
server_key_iv: "39487253dcaa3e98f1b377e08ff19c48"
server_key_version: "encryptionkey@0"
I20260928 03:04:58.220746 13041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:58.253458 13041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:58.258351 13041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:58.258602 13041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:58.277805 13041 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:41507
I20260928 03:04:58.277835 13107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:41507 every 8 connection(s)
I20260928 03:04:58.279258 13041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:04:58.283695 13108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:04:58.286851 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 13041
I20260928 03:04:58.287086 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:04:58.287869 6912 external_mini_cluster.cc:1428] Setting key bf030d435cc0dbbbee0504ce4c7e98ec
I20260928 03:04:58.297500 13108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9: Bootstrap starting.
I20260928 03:04:58.301590 13108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9: Neither blocks nor log segments found. Creating new log.
I20260928 03:04:58.303040 13108 log.cc:824] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9: Log is configured to *not* fsync() on all Append() calls
Sep 28 03:04:58 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM
I20260928 03:04:58.306603 13108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9: No bootstrap required, opened a new log
I20260928 03:04:58.313133 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:58.290234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.6.192.62:41507)
0928 03:04:58.291088 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:58.291100 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:58.291144 (+ 44us) server_negotiation.cc:407] Connection header received
0928 03:04:58.292901 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:58.292926 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:58.293366 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:58.293726 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:58.295766 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:58.297150 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:58.298953 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:58.299760 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:58.299946 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:58.305100 (+ 5154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:58.305109 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:58.305127 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:58.305165 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:58.308556 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:58.309454 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:58.309461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:58.309471 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:58.309643 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:58.310090 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:58.310095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:58.310098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:58.310702 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:58.311077 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:58.311565 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 03:04:58.312029 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":380,"threads_started":1}
I20260928 03:04:58.315121 13108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } }
I20260928 03:04:58.315574 13108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:04:58.315704 13108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a2716d3b7d545dd82f8fadedefd92d9, State: Initialized, Role: FOLLOWER
I20260928 03:04:58.316354 13108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } }
I20260928 03:04:58.316609 13108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:04:58.316748 13108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:04:58.316941 13108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:04:58.319190 13108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } }
I20260928 03:04:58.319765 13108 leader_election.cc:302] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a2716d3b7d545dd82f8fadedefd92d9; no voters:
I20260928 03:04:58.320376 13108 leader_election.cc:288] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:04:58.320546 13113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:04:58.321256 13113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [term 1 LEADER]: Becoming Leader. State: Replica: 7a2716d3b7d545dd82f8fadedefd92d9, State: Running, Role: LEADER
I20260928 03:04:58.321921 13113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } }
I20260928 03:04:58.323272 13108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:04:58.325445 13114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } }
I20260928 03:04:58.326061 13114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:58.325119 13115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a2716d3b7d545dd82f8fadedefd92d9. Latest consensus state: current_term: 1 leader_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } }
I20260928 03:04:58.326933 13115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: This master's current role is: LEADER
I20260928 03:04:58.328706 13121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:04:58.334359 13121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:04:58.344714 13121 catalog_manager.cc:1347] Generated new cluster ID: 3459c6c9524d4c0eab848e651e9c2e9c
I20260928 03:04:58.344870 13121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:04:58.375051 13121 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:04:58.376554 13121 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:04:58.384644 13121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9: Generated new TSK 0
I20260928 03:04:58.385715 13121 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.1.keytab.
Entry for principal oryx/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.1.keytab.
I20260928 03:04:58.452589 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41507
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.1.keytab
--principal=oryx/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:58.792927 13134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:58.793270 13134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:58.793366 13134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:58.803499 13134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:58.803673 13134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:58.803886 13134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:04:58.814751 13134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.1.keytab
--principal=oryx/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:58.816231 13134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:58.817580 13134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:58.832602 13139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:58.832635 13142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:58.832676 13140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:58.833658 13134 server_base.cc:1034] running on GCE node
I20260928 03:04:58.834295 13134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:58.835289 13134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:58.836551 13134 hybrid_clock.cc:648] HybridClock initialized: now 1790564698836505 us; error 44 us; skew 500 ppm
I20260928 03:04:58.836956 13134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:58 dist-test-slave-fjww krb5kdc[13028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564698, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:58.840503 13134 init.cc:377] Logged in from keytab as oryx/127.6.192.1@KRBTEST.COM (short username oryx)
I20260928 03:04:58.842185 13134 webserver.cc:460] Webserver started at http://127.6.192.1:41905/ using document root <none> and password file <none>
I20260928 03:04:58.842957 13134 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:58.843110 13134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:58.843430 13134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:58.846695 13134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "b258d7d74c2941b983bb7bfe5d709c7d"
format_stamp: "Formatted at 2026-09-28 03:04:58 on dist-test-slave-fjww"
server_key: "d50b2858fa7476d0fb01e971ffacecc9"
server_key_iv: "a4c73ad91e2f1b76347a73a8fb69e528"
server_key_version: "encryptionkey@0"
I20260928 03:04:58.847640 13134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "b258d7d74c2941b983bb7bfe5d709c7d"
format_stamp: "Formatted at 2026-09-28 03:04:58 on dist-test-slave-fjww"
server_key: "d50b2858fa7476d0fb01e971ffacecc9"
server_key_iv: "a4c73ad91e2f1b76347a73a8fb69e528"
server_key_version: "encryptionkey@0"
I20260928 03:04:58.853619 13134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 03:04:58.857918 13149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:58.859759 13134 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:04:58.859985 13134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "b258d7d74c2941b983bb7bfe5d709c7d"
format_stamp: "Formatted at 2026-09-28 03:04:58 on dist-test-slave-fjww"
server_key: "d50b2858fa7476d0fb01e971ffacecc9"
server_key_iv: "a4c73ad91e2f1b76347a73a8fb69e528"
server_key_version: "encryptionkey@0"
I20260928 03:04:58.860324 13134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:58.885738 13134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:58.890987 13134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:58.891340 13134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:58.892953 13134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:58.894650 13134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:58.894728 13134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:58.894816 13134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:58.894860 13134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:58.932201 13134 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:45339
I20260928 03:04:58.932302 13261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:45339 every 8 connection(s)
I20260928 03:04:58.933604 13134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:04:58.935913 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 13134
I20260928 03:04:58.936136 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:04:58.936939 6912 external_mini_cluster.cc:1428] Setting key ff210272d05e5cfad12bc35bd586c6e3
Sep 28 03:04:58 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564698, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.6.192.2@KRBTEST.COM; defaulting to no policy
I20260928 03:04:58.956867 13264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:58.936745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:34289 (local address 127.6.192.62:41507)
0928 03:04:58.937284 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:58.937292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:58.938539 (+ 1247us) server_negotiation.cc:407] Connection header received
0928 03:04:58.940750 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:58.940756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:58.940869 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:58.941009 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:58.942819 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:58.943677 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:58.945476 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:58.945929 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:58.946103 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:58.950447 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:58.950453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:58.950456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:58.950492 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:58.953286 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:58.954168 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:58.954176 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:58.954189 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:58.954320 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:58.954808 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:58.954815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:58.954818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:58.955099 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:58.955277 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:58.956180 (+ 903us) server_negotiation.cc:299] Negotiation successful
0928 03:04:58.956457 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":286,"threads_started":1}
I20260928 03:04:58.957998 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:58.937345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.1:34289)
0928 03:04:58.938366 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:58.938408 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:58.940424 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:58.941308 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:58.941319 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:58.941922 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:58.942596 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:58.942618 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:58.943867 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:58.943872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:58.945274 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:58.945289 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:58.946203 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:58.946211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:58.946487 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:58.947311 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:58.947338 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:58.950125 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:58.953481 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:58.953492 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:58.953514 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:58.954020 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:58.954444 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:58.954449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:58.954453 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:58.954677 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:58.955257 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:58.955266 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:58.955749 (+ 483us) client_negotiation.cc:769] Sending connection context
0928 03:04:58.956065 (+ 316us) client_negotiation.cc:240] Negotiation successful
0928 03:04:58.956670 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":417,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.192.2@KRBTEST.COM" created.
I20260928 03:04:58.959961 13262 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41507
I20260928 03:04:58.960496 13262 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:58.961589 13262 heartbeater.cc:511] Master 127.6.192.62:41507 requested a full tablet report, sending...
I20260928 03:04:58.964306 13073 ts_manager.cc:194] Registered new tserver with Master: b258d7d74c2941b983bb7bfe5d709c7d (127.6.192.1:45339)
I20260928 03:04:58.965847 13073 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:34289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.2.keytab.
Entry for principal oryx/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.2.keytab.
I20260928 03:04:58.987784 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41507
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.2.keytab
--principal=oryx/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:59.327651 13268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:59.327996 13268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:59.328178 13268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:59.335794 13268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:59.335954 13268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:59.336136 13268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:04:59.346393 13268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.2.keytab
--principal=oryx/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:59.347920 13268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:59.349395 13268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:59.364414 13274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:59.364600 13276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:59.364635 13273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:59.365475 13268 server_base.cc:1034] running on GCE node
I20260928 03:04:59.366149 13268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:59.367156 13268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:59.368422 13268 hybrid_clock.cc:648] HybridClock initialized: now 1790564699368294 us; error 61 us; skew 500 ppm
I20260928 03:04:59.368781 13268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:59 dist-test-slave-fjww krb5kdc[13028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564699, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:59.372234 13268 init.cc:377] Logged in from keytab as oryx/127.6.192.2@KRBTEST.COM (short username oryx)
I20260928 03:04:59.374001 13268 webserver.cc:460] Webserver started at http://127.6.192.2:38199/ using document root <none> and password file <none>
I20260928 03:04:59.374747 13268 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:59.374858 13268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:59.375123 13268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:59.378322 13268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "b73e7c5e06f7404c928ec74316dd4512"
format_stamp: "Formatted at 2026-09-28 03:04:59 on dist-test-slave-fjww"
server_key: "96e20727adf81484800233cb5aeb4c7e"
server_key_iv: "c7153f52d641ec549fe2eca357bada94"
server_key_version: "encryptionkey@0"
I20260928 03:04:59.379118 13268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "b73e7c5e06f7404c928ec74316dd4512"
format_stamp: "Formatted at 2026-09-28 03:04:59 on dist-test-slave-fjww"
server_key: "96e20727adf81484800233cb5aeb4c7e"
server_key_iv: "c7153f52d641ec549fe2eca357bada94"
server_key_version: "encryptionkey@0"
I20260928 03:04:59.385329 13268 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 03:04:59.389796 13283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:59.391710 13268 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260928 03:04:59.391911 13268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "b73e7c5e06f7404c928ec74316dd4512"
format_stamp: "Formatted at 2026-09-28 03:04:59 on dist-test-slave-fjww"
server_key: "96e20727adf81484800233cb5aeb4c7e"
server_key_iv: "c7153f52d641ec549fe2eca357bada94"
server_key_version: "encryptionkey@0"
I20260928 03:04:59.392160 13268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:59.417615 13268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:59.422374 13268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:59.422636 13268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:59.423660 13268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:59.425168 13268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:59.425244 13268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:59.425388 13268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:59.425436 13268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:59.462570 13268 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:36405
I20260928 03:04:59.462698 13395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:36405 every 8 connection(s)
I20260928 03:04:59.464015 13268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:04:59.471531 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 13268
I20260928 03:04:59.471737 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:04:59.472200 6912 external_mini_cluster.cc:1428] Setting key bcc82d0d87d23eaeaa2819e170c16654
Sep 28 03:04:59 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564699, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM
I20260928 03:04:59.488977 13398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:59.467034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:36455 (local address 127.6.192.62:41507)
0928 03:04:59.467607 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:59.467612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:59.468680 (+ 1068us) server_negotiation.cc:407] Connection header received
0928 03:04:59.470790 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:59.470795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:59.470882 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:59.471013 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:59.473163 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:59.474151 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:59.476168 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:59.476964 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:59.477116 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:59.481188 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:04:59.481195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:04:59.481199 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:04:59.481250 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:04:59.485166 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:59.486076 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:59.486085 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:59.486092 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:59.486254 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:04:59.486725 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:04:59.486733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:04:59.486739 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:04:59.487133 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:04:59.487292 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 03:04:59.488163 (+ 871us) server_negotiation.cc:299] Negotiation successful
0928 03:04:59.488474 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":246,"threads_started":1}
I20260928 03:04:59.489835 13399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:59.467677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.2:36455)
0928 03:04:59.468491 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:59.468539 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:59.470465 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:59.471254 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:59.471263 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:59.472094 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:59.472871 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:59.472902 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:59.474333 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:59.474340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:59.475990 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:59.476005 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:59.477212 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:59.477218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:59.477484 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:59.478303 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:59.478330 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:04:59.480867 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:04:59.485350 (+ 4483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:59.485359 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:59.485376 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:59.485921 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:59.486383 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:04:59.486387 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:04:59.486390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:04:59.486598 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:04:59.487279 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:04:59.487287 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:04:59.487727 (+ 440us) client_negotiation.cc:769] Sending connection context
0928 03:04:59.488021 (+ 294us) client_negotiation.cc:240] Negotiation successful
0928 03:04:59.488523 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":311,"threads_started":1}
I20260928 03:04:59.491884 13396 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41507
I20260928 03:04:59.492229 13396 heartbeater.cc:464] Registering TS with master...
I20260928 03:04:59.493026 13396 heartbeater.cc:511] Master 127.6.192.62:41507 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.6.192.3@KRBTEST.COM; defaulting to no policy
I20260928 03:04:59.494761 13073 ts_manager.cc:194] Registered new tserver with Master: b73e7c5e06f7404c928ec74316dd4512 (127.6.192.2:36405)
I20260928 03:04:59.495724 13073 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:36455
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.192.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.3.keytab.
Entry for principal oryx/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.3.keytab.
I20260928 03:04:59.522985 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41507
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.3.keytab
--principal=oryx/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:04:59.861759 13402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:04:59.862061 13402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:04:59.862155 13402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:04:59.869946 13402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:04:59.870071 13402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:04:59.870188 13402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:04:59.880226 13402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.3.keytab
--principal=oryx/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:04:59.881556 13402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:04:59.882846 13402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:04:59.896730 13408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:59.896684 13407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:04:59.896768 13410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:04:59.898006 13402 server_base.cc:1034] running on GCE node
I20260928 03:04:59.898695 13402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:04:59.899708 13402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:04:59.900942 13402 hybrid_clock.cc:648] HybridClock initialized: now 1790564699900896 us; error 35 us; skew 500 ppm
I20260928 03:04:59.901299 13402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:04:59 dist-test-slave-fjww krb5kdc[13028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564699, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:04:59.904799 13402 init.cc:377] Logged in from keytab as oryx/127.6.192.3@KRBTEST.COM (short username oryx)
I20260928 03:04:59.906562 13402 webserver.cc:460] Webserver started at http://127.6.192.3:45681/ using document root <none> and password file <none>
I20260928 03:04:59.907270 13402 fs_manager.cc:359] Metadata directory not provided
I20260928 03:04:59.907379 13402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:04:59.907697 13402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:04:59.910702 13402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "4af241c0395a4b3aa8178f812e35329c"
format_stamp: "Formatted at 2026-09-28 03:04:59 on dist-test-slave-fjww"
server_key: "5d0b2fd30dc6231a77dc7fc83317e40c"
server_key_iv: "7eaba3fe5a0264cc14aede74bc6064b6"
server_key_version: "encryptionkey@0"
I20260928 03:04:59.911542 13402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "4af241c0395a4b3aa8178f812e35329c"
format_stamp: "Formatted at 2026-09-28 03:04:59 on dist-test-slave-fjww"
server_key: "5d0b2fd30dc6231a77dc7fc83317e40c"
server_key_iv: "7eaba3fe5a0264cc14aede74bc6064b6"
server_key_version: "encryptionkey@0"
I20260928 03:04:59.917083 13402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 03:04:59.921026 13417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:59.922670 13402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 03:04:59.922870 13402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "4af241c0395a4b3aa8178f812e35329c"
format_stamp: "Formatted at 2026-09-28 03:04:59 on dist-test-slave-fjww"
server_key: "5d0b2fd30dc6231a77dc7fc83317e40c"
server_key_iv: "7eaba3fe5a0264cc14aede74bc6064b6"
server_key_version: "encryptionkey@0"
I20260928 03:04:59.923100 13402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:04:59.939152 13402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:04:59.944195 13402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:04:59.944535 13402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:04:59.945550 13402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:04:59.947017 13402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:04:59.947119 13402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:59.947263 13402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:04:59.947336 13402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:04:59.969682 13262 heartbeater.cc:503] Master 127.6.192.62:41507 was elected leader, sending a full tablet report...
I20260928 03:04:59.983593 13402 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:44019
I20260928 03:04:59.983686 13529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:44019 every 8 connection(s)
I20260928 03:04:59.984906 13402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:04:59.986253 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 13402
I20260928 03:04:59.986444 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:04:59.987043 6912 external_mini_cluster.cc:1428] Setting key 772105f927ec09305df655e2193dce26
Sep 28 03:04:59 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564699, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM
I20260928 03:05:00.009871 13398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:59.988106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:33137 (local address 127.6.192.62:41507)
0928 03:04:59.988302 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 03:04:59.988308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:04:59.989841 (+ 1533us) server_negotiation.cc:407] Connection header received
0928 03:04:59.992193 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:04:59.992200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:04:59.992309 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:04:59.992431 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:04:59.994153 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:59.995018 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:59.996939 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:04:59.997808 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:04:59.998172 (+ 364us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:00.002661 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:00.002667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:00.002670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:00.002707 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:00.006067 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:00.007092 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:00.007098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:00.007100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:00.007257 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:00.007856 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:00.007861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:00.007863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:00.008185 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:00.008382 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:00.009226 (+ 844us) server_negotiation.cc:299] Negotiation successful
0928 03:05:00.009495 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 03:05:00.010985 13532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:04:59.988623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.3:33137)
0928 03:04:59.989663 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0928 03:04:59.989717 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 03:04:59.991886 (+ 2169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:04:59.992687 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:04:59.992696 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:04:59.993299 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:04:59.993959 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:59.993980 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:59.995214 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:59.995218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:59.996777 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:04:59.996790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:04:59.998292 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:04:59.998300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:04:59.998638 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:04:59.999427 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:04:59.999524 (+ 97us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:00.002413 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:00.006289 (+ 3876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:00.006302 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:00.006325 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:00.006920 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:00.007389 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:00.007395 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:00.007399 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:00.007706 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:00.008348 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:00.008358 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:00.008778 (+ 420us) client_negotiation.cc:769] Sending connection context
0928 03:05:00.009089 (+ 311us) client_negotiation.cc:240] Negotiation successful
0928 03:05:00.009731 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":388,"threads_started":1}
I20260928 03:05:00.012940 13530 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41507
I20260928 03:05:00.013340 13530 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:00.014336 13530 heartbeater.cc:511] Master 127.6.192.62:41507 requested a full tablet report, sending...
I20260928 03:05:00.016474 13073 ts_manager.cc:194] Registered new tserver with Master: 4af241c0395a4b3aa8178f812e35329c (127.6.192.3:44019)
I20260928 03:05:00.017477 13073 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:33137
I20260928 03:05:00.029919 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.192.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.192.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.192.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.61.keytab.
Entry for principal oryx/127.6.192.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.61.keytab.
I20260928 03:05:00.084247 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.61:41467
--webserver_interface=127.6.192.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.61.keytab
--principal=oryx/127.6.192.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:41507,127.6.192.61:41467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:05:00.435411 13535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:00.435820 13535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:00.435987 13535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:00.443570 13535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:05:00.443686 13535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:00.443789 13535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:00.443893 13535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:05:00.443982 13535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:05:00.454082 13535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:41507,127.6.192.61:41467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.61.keytab
--principal=oryx/127.6.192.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.61:41467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.192.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
W20260928 03:05:00.454509 13535 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.192.62:41507,127.6.192.61:41467'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 03:05:00.455698 13535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:00.457094 13535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:00.470791 13543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:00.470865 13541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:00.470877 13540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:00.471652 13535 server_base.cc:1034] running on GCE node
I20260928 03:05:00.472554 13535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:00.474090 13535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:00.475392 13535 hybrid_clock.cc:648] HybridClock initialized: now 1790564700475315 us; error 60 us; skew 500 ppm
I20260928 03:05:00.475847 13535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:00 dist-test-slave-fjww krb5kdc[13028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564700, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:00.484692 13535 init.cc:377] Logged in from keytab as oryx/127.6.192.61@KRBTEST.COM (short username oryx)
I20260928 03:05:00.486537 13535 webserver.cc:460] Webserver started at http://127.6.192.61:41961/ using document root <none> and password file <none>
I20260928 03:05:00.487319 13535 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:00.487417 13535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:00.487841 13535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:00.491137 13535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/data/instance:
uuid: "569125e0394b44a8a3d58d5fb58d6821"
format_stamp: "Formatted at 2026-09-28 03:05:00 on dist-test-slave-fjww"
server_key: "60333392c7f06fb03ceab728b14b859b"
server_key_iv: "1abe6b85630930e98f80e58a04ccd7a3"
server_key_version: "encryptionkey@0"
I20260928 03:05:00.492055 13535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/wal/instance:
uuid: "569125e0394b44a8a3d58d5fb58d6821"
format_stamp: "Formatted at 2026-09-28 03:05:00 on dist-test-slave-fjww"
server_key: "60333392c7f06fb03ceab728b14b859b"
server_key_iv: "1abe6b85630930e98f80e58a04ccd7a3"
server_key_version: "encryptionkey@0"
I20260928 03:05:00.497864 13535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 03:05:00.498826 13396 heartbeater.cc:503] Master 127.6.192.62:41507 was elected leader, sending a full tablet report...
I20260928 03:05:00.502425 13550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:00.504264 13535 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:05:00.504487 13535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/wal
uuid: "569125e0394b44a8a3d58d5fb58d6821"
format_stamp: "Formatted at 2026-09-28 03:05:00 on dist-test-slave-fjww"
server_key: "60333392c7f06fb03ceab728b14b859b"
server_key_iv: "1abe6b85630930e98f80e58a04ccd7a3"
server_key_version: "encryptionkey@0"
I20260928 03:05:00.504810 13535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:00.522675 13535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:00.528280 13535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:00.528584 13535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:00.550060 13535 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.61:41467
I20260928 03:05:00.550068 13601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.61:41467 every 8 connection(s)
I20260928 03:05:00.551626 13535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/data/info.pb
I20260928 03:05:00.555785 13602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:00.557134 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 13535
I20260928 03:05:00.565615 13602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:05:00 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.192.61@KRBTEST.COM
Sep 28 03:05:00 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564700, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.61@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM
I20260928 03:05:00.581882 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:00.559580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.6.192.61:41467)
0928 03:05:00.560717 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:00.560729 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:00.560775 (+ 46us) server_negotiation.cc:407] Connection header received
0928 03:05:00.562519 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:00.562543 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:00.562961 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:00.563402 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:00.564957 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.566309 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.567636 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.568485 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.568680 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:00.574069 (+ 5389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:00.574077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:00.574088 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:00.574127 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:00.577627 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:00.578424 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:00.578431 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:00.578438 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:00.578592 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:00.578998 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:00.579009 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:00.579011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:00.579607 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:00.579934 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:00.580370 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 03:05:00.580810 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":727,"threads_started":1}
I20260928 03:05:00.584302 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:00.568270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:35348)
0928 03:05:00.569238 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:00.569280 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:00.569453 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:00.569890 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:00.569895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:00.570522 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:00.570769 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:00.570777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.572375 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.572382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:00.573802 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:00.573814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.574566 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.574578 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:00.574762 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:00.575530 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:00.575569 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:00.578328 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:00.581556 (+ 3228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:00.581567 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:00.581575 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:00.582078 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:00.582450 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:00.582455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:00.582459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:00.582589 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:00.583246 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:00.583252 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:00.583551 (+ 299us) client_negotiation.cc:769] Sending connection context
0928 03:05:00.583744 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 03:05:00.584006 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":562,"threads_started":1}
I20260928 03:05:00.584455 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:00.568288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.6.192.62:41507)
0928 03:05:00.568833 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:00.568841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:00.569410 (+ 569us) server_negotiation.cc:407] Connection header received
0928 03:05:00.569588 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:00.569595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:00.569699 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:00.569794 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:00.570955 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.572200 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.573976 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.574380 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.574474 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:00.578544 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:00.578549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:00.578552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:00.578586 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:00.581410 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:00.582199 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:00.582205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:00.582209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:00.582335 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:00.582724 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:00.582729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:00.582732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:00.583104 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:00.583265 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:00.583862 (+ 597us) server_negotiation.cc:299] Negotiation successful
0928 03:05:00.584107 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":251,"threads_started":1}
I20260928 03:05:00.586072 13602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:05:00 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564700, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.61@KRBTEST.COM for oryx/127.6.192.61@KRBTEST.COM
I20260928 03:05:00.599058 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:00.586710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:41467 (local address 127.0.0.1:40326)
0928 03:05:00.586903 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:00.586918 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:00.586999 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:00.587381 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:00.587384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:00.587753 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:00.587895 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:00.587904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.588922 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.588925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:00.589901 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:00.589912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.590655 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.590660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:00.590824 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:00.591729 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:00.591745 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:00.593803 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:00.596971 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:00.596975 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:00.596981 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:00.597388 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:00.597728 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:00.597732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:00.597734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:00.597872 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:00.598340 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:00.598344 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:00.598473 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 03:05:00.598597 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 03:05:00.598819 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 03:05:00.599687 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:00.586825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40326 (local address 127.6.192.61:41467)
0928 03:05:00.586987 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:00.586998 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:00.587023 (+ 25us) server_negotiation.cc:407] Connection header received
0928 03:05:00.587067 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:00.587070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:00.587121 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:00.587307 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:00.588025 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.588784 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.590038 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.590512 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.590618 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:00.594006 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:00.594011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:00.594014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:00.594043 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:00.596803 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:00.597495 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:00.597499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:00.597501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:00.597619 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:00.597965 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:00.597968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:00.597970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:00.598237 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:00.598393 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:00.598695 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 03:05:00.599319 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":66}
I20260928 03:05:00.604965 13602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: Bootstrap starting.
I20260928 03:05:00.608583 13602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:00.609861 13602 log.cc:824] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: Log is configured to *not* fsync() on all Append() calls
I20260928 03:05:00.613068 13602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: No bootstrap required, opened a new log
I20260928 03:05:00.621328 13602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } }
I20260928 03:05:00.621800 13602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:00.621923 13602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 569125e0394b44a8a3d58d5fb58d6821, State: Initialized, Role: FOLLOWER
I20260928 03:05:00.622589 13602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } }
I20260928 03:05:00.623736 13609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } } }
I20260928 03:05:00.624063 13609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:05:00.625000 13602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 03:05:00.635282 13620 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 03:05:00.635460 13620 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 03:05:00.638357 13535 master_runner.cc:428] Detected that this master 569125e0394b44a8a3d58d5fb58d6821 is joining an existing cluster
I20260928 03:05:00.638492 13535 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.192.61:41467
WARNING: no policy specified for oryx/127.6.192.60@KRBTEST.COM; defaulting to no policy
I20260928 03:05:00.666296 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:00.647868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.6.192.61:41467)
0928 03:05:00.648054 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:00.648059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:00.648587 (+ 528us) server_negotiation.cc:407] Connection header received
0928 03:05:00.648825 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:00.648832 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:00.648912 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:00.648987 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:00.650515 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.651797 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.653988 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.654787 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.654918 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:00.657939 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:00.657948 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:00.657955 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:00.657998 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:00.662596 (+ 4598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:00.663684 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:00.663689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:00.663692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:00.663839 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:00.664335 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:00.664342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:00.664345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:00.664783 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:00.664946 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:00.665396 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 03:05:00.665844 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":38}
I20260928 03:05:00.666321 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:00.647818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:41467 (local address 127.0.0.1:40342)
0928 03:05:00.648476 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:00.648495 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:00.648684 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:00.649194 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:00.649201 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:00.650022 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:00.650371 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:00.650386 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.652044 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.652052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:00.653843 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:00.653860 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.654998 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.655010 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:00.655264 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:00.656324 (+ 1060us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:00.656352 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:00.657744 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:00.662797 (+ 5053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:00.662802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:00.662807 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:00.663556 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:00.664009 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:00.664019 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:00.664022 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:00.664216 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:00.665030 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:00.665034 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:00.665158 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 03:05:00.665309 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 03:05:00.665673 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":26,"thread_start_us":326,"threads_started":1}
I20260928 03:05:00.666903 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:00.646848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.6.192.62:41507)
0928 03:05:00.647071 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:00.647080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:00.647773 (+ 693us) server_negotiation.cc:407] Connection header received
0928 03:05:00.648105 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:00.648112 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:00.648212 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:00.648291 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:00.650003 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.651210 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.653420 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.654147 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.654257 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:00.657924 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:00.657931 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:00.657936 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:00.657981 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:00.661766 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:00.663470 (+ 1704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:00.663477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:00.663481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:00.664502 (+ 1021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:00.664971 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:00.664977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:00.664980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:00.665361 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:00.665521 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:00.666254 (+ 733us) server_negotiation.cc:299] Negotiation successful
0928 03:05:00.666533 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.192.60@KRBTEST.COM" created.
I20260928 03:05:00.667960 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:00.646853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:35356)
0928 03:05:00.647642 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:00.647665 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:00.647919 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:00.648562 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:00.648580 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:00.649292 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:00.649796 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:00.649810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.651402 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.651408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:00.653231 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:00.653246 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.654593 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.654600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:00.655382 (+ 782us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:00.656179 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:00.656210 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:00.657661 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:00.661974 (+ 4313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:00.661979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:00.661984 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:00.663278 (+ 1294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:00.664656 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:00.664660 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:00.664664 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:00.664830 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:00.665583 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:00.665588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:00.665924 (+ 336us) client_negotiation.cc:769] Sending connection context
0928 03:05:00.667087 (+ 1163us) client_negotiation.cc:240] Negotiation successful
0928 03:05:00.667424 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":72,"thread_start_us":396,"threads_started":1}
Sep 28 03:05:00 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564698, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.62@KRBTEST.COM for oryx/127.6.192.61@KRBTEST.COM
I20260928 03:05:00.694984 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:00.674821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.6.192.61:41467)
0928 03:05:00.675019 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:00.675027 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:00.676192 (+ 1165us) server_negotiation.cc:407] Connection header received
0928 03:05:00.676623 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:00.676630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:00.676746 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:00.676868 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:00.678489 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.679635 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.681850 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.682598 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.682736 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:00.687543 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:00.687550 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:00.687555 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:00.687603 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:00.691254 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:00.692468 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:00.692473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:00.692476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:00.692691 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:00.693160 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:00.693165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:00.693175 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:00.693556 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:00.693727 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:00.694289 (+ 562us) server_negotiation.cc:299] Negotiation successful
0928 03:05:00.694588 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 03:05:00.695052 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:00.674755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:41467 (local address 127.0.0.1:40352)
0928 03:05:00.676030 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:00.676076 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:00.676364 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:00.676936 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:00.676948 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:00.677939 (+ 991us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:00.678275 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:00.678291 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.679784 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.679793 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:00.681662 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:00.681682 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.682804 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.682813 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:00.683078 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:00.683978 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:00.684006 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:00.687238 (+ 3232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:00.691434 (+ 4196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:00.691490 (+ 56us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:00.691499 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:00.692311 (+ 812us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:00.692820 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:00.692826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:00.692829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:00.693027 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:00.693704 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:00.693713 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:00.693911 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 03:05:00.694187 (+ 276us) client_negotiation.cc:240] Negotiation successful
0928 03:05:00.694556 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"thread_start_us":849,"threads_started":1}
I20260928 03:05:00.697557 13073 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.192.61:41467: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821"
member_type: NON_VOTER
last_known_addr {
host: "127.6.192.61"
port: 41467
}
attrs {
promote: true
}
}
dest_uuid: "7a2716d3b7d545dd82f8fadedefd92d9"
cas_config_opid_index: -1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.192.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.60.keytab.
Entry for principal oryx/127.6.192.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.60.keytab.
I20260928 03:05:00.698760 13073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: NON_VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: true } }
I20260928 03:05:00.699605 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.60:40419
--webserver_interface=127.6.192.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.60.keytab
--principal=oryx/127.6.192.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:41507,127.6.192.61:41467,127.6.192.60:40419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:05:00.711508 13577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:00.711002 13633 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 569125e0394b44a8a3d58d5fb58d6821 (127.6.192.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: NON_VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: true } } }
I20260928 03:05:00.715646 13577 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 1 FOLLOWER]: Refusing update from remote peer 7a2716d3b7d545dd82f8fadedefd92d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 03:05:00.716223 13633 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.192.61:41467
I20260928 03:05:00.717216 13535 master.cc:537] Master@127.6.192.61:41467 shutting down...
I20260928 03:05:00.716799 13633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: NON_VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: true } } }
I20260928 03:05:00.718274 13633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: This master's current role is: LEADER
I20260928 03:05:00.720747 13633 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: NON_VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 03:05:00.726455 13059 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260928 03:05:00.729138 13535 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:05:00.729652 13535 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:00.729885 13535 tablet_replica.cc:330] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: stopping tablet replica
W20260928 03:05:00.729950 13059 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 -> Peer 569125e0394b44a8a3d58d5fb58d6821 (127.6.192.61:41467): Couldn't send request to peer 569125e0394b44a8a3d58d5fb58d6821. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 03:05:00.737268 13535 master.cc:552] Master@127.6.192.61:41467 shutdown complete.
I20260928 03:05:00.737375 13535 master_runner.cc:305] Clearing existing system tablet
I20260928 03:05:00.737802 13535 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:00.737900 13535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:00.744277 13535 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.003s
I20260928 03:05:00.746064 13638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:00.746454 13535 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:00.746757 13535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260928 03:05:00.746843 13535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/wal
uuid: "569125e0394b44a8a3d58d5fb58d6821"
format_stamp: "Formatted at 2026-09-28 03:05:00 on dist-test-slave-fjww"
server_key: "60333392c7f06fb03ceab728b14b859b"
server_key_iv: "1abe6b85630930e98f80e58a04ccd7a3"
server_key_version: "encryptionkey@0"
I20260928 03:05:00.752154 13535 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:00.759562 13535 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 03:05:00.759788 13535 log.cc:1196] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 03:05:00.760246 13535 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: Deleting consensus metadata
I20260928 03:05:00.760519 13535 master_runner.cc:315] Copying system tablet from 127.6.192.62:41507
I20260928 03:05:00.764964 13535 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: tablet copy: Beginning tablet copy session from remote peer at address 127.6.192.62:41507
I20260928 03:05:00.782749 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:00.765804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.6.192.62:41507)
0928 03:05:00.766045 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:00.766055 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:00.766536 (+ 481us) server_negotiation.cc:407] Connection header received
0928 03:05:00.766721 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:00.766729 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:00.766843 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:00.766965 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:00.768166 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.769333 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.771419 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.772129 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.772282 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:00.774695 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:00.774702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:00.774707 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:00.774755 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:00.779166 (+ 4411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:00.780248 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:00.780257 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:00.780263 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:00.780477 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:00.780989 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:00.780996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:00.781001 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:00.781417 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:00.781582 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:00.782090 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 03:05:00.782372 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 03:05:00.783228 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:00.765764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:35370)
0928 03:05:00.766395 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:00.766416 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:00.766580 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:00.767018 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:00.767023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:00.767634 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:00.767972 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:00.767986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.769540 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.769548 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:00.771165 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:00.771186 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:00.772330 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:00.772338 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:00.772503 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:00.773267 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:00.773298 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:00.774422 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:00.779405 (+ 4983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:00.779412 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:00.779417 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:00.780062 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:00.780630 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:00.780636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:00.780640 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:00.780833 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:00.781566 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:00.781571 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:00.781784 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 03:05:00.781964 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 03:05:00.782344 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":367,"threads_started":1}
I20260928 03:05:00.784426 13093 tablet_copy_service.cc:140] P 7a2716d3b7d545dd82f8fadedefd92d9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 569125e0394b44a8a3d58d5fb58d6821 ({username='oryx', principal='oryx/127.6.192.61@KRBTEST.COM'} at 127.0.0.1:35370)
I20260928 03:05:00.784700 13093 tablet_copy_service.cc:161] P 7a2716d3b7d545dd82f8fadedefd92d9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 569125e0394b44a8a3d58d5fb58d6821 at {username='oryx', principal='oryx/127.6.192.61@KRBTEST.COM'} at 127.0.0.1:35370: session id = 569125e0394b44a8a3d58d5fb58d6821-00000000000000000000000000000000
I20260928 03:05:00.789494 13093 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 03:05:00.792311 13535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 03:05:00.799472 13535 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: tablet copy: Starting download of 0 data blocks...
I20260928 03:05:00.799837 13535 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: tablet copy: Starting download of 1 WAL segments...
I20260928 03:05:00.802975 13535 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 03:05:00.807998 13093 tablet_copy_service.cc:342] P 7a2716d3b7d545dd82f8fadedefd92d9: Request end of tablet copy session 569125e0394b44a8a3d58d5fb58d6821-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.192.61@KRBTEST.COM'} at 127.0.0.1:35370
I20260928 03:05:00.808269 13093 tablet_copy_service.cc:433] P 7a2716d3b7d545dd82f8fadedefd92d9: ending tablet copy session 569125e0394b44a8a3d58d5fb58d6821-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 569125e0394b44a8a3d58d5fb58d6821
I20260928 03:05:00.821471 13535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:00.824491 13645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:00.825860 13646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:00.833264 13648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:00.837163 13535 server_base.cc:1034] running on GCE node
I20260928 03:05:00.837709 13535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:00.838491 13535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:00.839721 13535 hybrid_clock.cc:648] HybridClock initialized: now 1790564700839639 us; error 52 us; skew 500 ppm
I20260928 03:05:00.840009 13535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:00 dist-test-slave-fjww krb5kdc[13028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564700, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:00.842342 13535 init.cc:377] Logged in from keytab as oryx/127.6.192.61@KRBTEST.COM (short username oryx)
I20260928 03:05:00.843982 13535 webserver.cc:460] Webserver started at http://127.6.192.61:46235/ using document root <none> and password file <none>
I20260928 03:05:00.844398 13535 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:00.844487 13535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:00.848040 13535 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 03:05:00.850127 13655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:00.850638 13535 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 03:05:00.850843 13535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/wal
uuid: "569125e0394b44a8a3d58d5fb58d6821"
format_stamp: "Formatted at 2026-09-28 03:05:00 on dist-test-slave-fjww"
server_key: "60333392c7f06fb03ceab728b14b859b"
server_key_iv: "1abe6b85630930e98f80e58a04ccd7a3"
server_key_version: "encryptionkey@0"
I20260928 03:05:00.851114 13535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:00.876626 13535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:00.881421 13535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:00.907226 13535 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.61:41467
I20260928 03:05:00.907256 13706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.61:41467 every 8 connection(s)
I20260928 03:05:00.908416 13535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-1/data/info.pb
I20260928 03:05:00.916430 13707 sys_catalog.cc:263] Verifying existing consensus state
I20260928 03:05:00.920502 13707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: Bootstrap starting.
I20260928 03:05:00.940274 13707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 03:05:00.940857 13707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: Bootstrap complete.
I20260928 03:05:00.942080 13707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: NON_VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: true } }
I20260928 03:05:00.942436 13707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 569125e0394b44a8a3d58d5fb58d6821, State: Initialized, Role: LEARNER
I20260928 03:05:00.942687 13707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: NON_VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: true } }
I20260928 03:05:00.943887 13707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:05:00.943734 13710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: NON_VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: true } } }
I20260928 03:05:00.944198 13710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [sys.catalog]: This master's current role is: LEARNER
I20260928 03:05:00.955499 13721 catalog_manager.cc:1259] Loaded cluster ID: 3459c6c9524d4c0eab848e651e9c2e9c
I20260928 03:05:00.955627 13721 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: loading cluster ID for follower catalog manager: success
I20260928 03:05:00.959507 13721 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: acquiring CA information for follower catalog manager: success
I20260928 03:05:00.962363 13721 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 03:05:01.020429 13530 heartbeater.cc:503] Master 127.6.192.62:41507 was elected leader, sending a full tablet report...
W20260928 03:05:01.161098 13634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:01.161505 13634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:01.161655 13634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:01.169503 13634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:05:01.169625 13634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:01.169672 13634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:01.169730 13634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:05:01.169775 13634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:05:01.180119 13634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:41507,127.6.192.61:41467,127.6.192.60:40419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.60.keytab
--principal=oryx/127.6.192.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.60:40419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.192.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:01.181447 13634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:01.182703 13634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:01.195147 13727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:01.195230 13726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:01.195154 13729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:01.196012 13634 server_base.cc:1034] running on GCE node
I20260928 03:05:01.196985 13634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:01.198537 13634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:01.199790 13634 hybrid_clock.cc:648] HybridClock initialized: now 1790564701199730 us; error 57 us; skew 500 ppm
I20260928 03:05:01.200286 13634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:01 dist-test-slave-fjww krb5kdc[13028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564701, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:01.203820 13634 init.cc:377] Logged in from keytab as oryx/127.6.192.60@KRBTEST.COM (short username oryx)
I20260928 03:05:01.205575 13634 webserver.cc:460] Webserver started at http://127.6.192.60:35143/ using document root <none> and password file <none>
I20260928 03:05:01.206300 13634 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:01.206449 13634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:01.206756 13634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:01.209952 13634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/data/instance:
uuid: "04b9a59f6fd94c6892c5278980919e61"
format_stamp: "Formatted at 2026-09-28 03:05:01 on dist-test-slave-fjww"
server_key: "64fbe2376e5ebffdfaef9d5d67cf31df"
server_key_iv: "cba08deba4e9bb9f376fd7f3ea593dac"
server_key_version: "encryptionkey@0"
I20260928 03:05:01.210793 13634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/wal/instance:
uuid: "04b9a59f6fd94c6892c5278980919e61"
format_stamp: "Formatted at 2026-09-28 03:05:01 on dist-test-slave-fjww"
server_key: "64fbe2376e5ebffdfaef9d5d67cf31df"
server_key_iv: "cba08deba4e9bb9f376fd7f3ea593dac"
server_key_version: "encryptionkey@0"
I20260928 03:05:01.216897 13634 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 03:05:01.220690 13736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:01.222393 13634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 03:05:01.222609 13634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/wal
uuid: "04b9a59f6fd94c6892c5278980919e61"
format_stamp: "Formatted at 2026-09-28 03:05:01 on dist-test-slave-fjww"
server_key: "64fbe2376e5ebffdfaef9d5d67cf31df"
server_key_iv: "cba08deba4e9bb9f376fd7f3ea593dac"
server_key_version: "encryptionkey@0"
I20260928 03:05:01.222844 13634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:01.245339 13634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:01.250013 13634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:01.250403 13634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:01.271610 13634 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.60:40419
I20260928 03:05:01.271703 13787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.60:40419 every 8 connection(s)
I20260928 03:05:01.273188 13634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/data/info.pb
I20260928 03:05:01.276609 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 13634
I20260928 03:05:01.277761 13788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:01.291582 13788 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:05:01 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.192.60@KRBTEST.COM
Sep 28 03:05:01 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564701, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.60@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM
I20260928 03:05:01.324258 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.309407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47080 (local address 127.6.192.61:41467)
0928 03:05:01.314236 (+ 4829us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:01.314243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:01.316241 (+ 1998us) server_negotiation.cc:407] Connection header received
0928 03:05:01.316474 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:01.316480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:01.316584 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:01.316718 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:01.317495 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.318745 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.321385 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.322852 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.323031 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.323301 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:01.323686 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 03:05:01.323855 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4672,"thread_start_us":4540,"threads_started":1}
I20260928 03:05:01.324294 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.310915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:41467 (local address 127.0.0.1:47080)
0928 03:05:01.316093 (+ 5178us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:01.316116 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:01.316283 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:01.316805 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:01.316809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:01.316861 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:01.317279 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.317293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.318904 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.318911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.321055 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.321079 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.323084 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.323092 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.323336 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.323516 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 03:05:01.323655 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 03:05:01.323859 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5005,"thread_start_us":4706,"threads_started":1}
I20260928 03:05:01.325721 13682 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260928 03:05:01.326669 13710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a2716d3b7d545dd82f8fadedefd92d9. Latest consensus state: current_term: 1 leader_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: NON_VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: true } } }
I20260928 03:05:01.327008 13710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [sys.catalog]: This master's current role is: LEARNER
I20260928 03:05:01.328811 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.279125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.6.192.60:40419)
0928 03:05:01.296592 (+ 17467us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:01.296616 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:01.296694 (+ 78us) server_negotiation.cc:407] Connection header received
0928 03:05:01.300382 (+ 3688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:01.300433 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:01.301269 (+ 836us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:01.301868 (+ 599us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:01.305346 (+ 3478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.307618 (+ 2272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.309699 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.311362 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.311679 (+ 317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.318308 (+ 6629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:01.318318 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:01.318332 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:01.318375 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:01.322255 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.323374 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.323384 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.323395 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.323658 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.324725 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.324745 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.324749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.325566 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:01.325978 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:01.326454 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 03:05:01.327084 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17090,"thread_start_us":16745,"threads_started":1}
I20260928 03:05:01.330658 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.307697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51114 (local address 127.6.192.62:41507)
0928 03:05:01.309173 (+ 1476us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:01.309179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:01.309201 (+ 22us) server_negotiation.cc:407] Connection header received
0928 03:05:01.309270 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:01.309276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:01.309348 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:01.309464 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:01.312064 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.313703 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.316400 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.317147 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.317318 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.322815 (+ 5497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:01.322823 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:01.322828 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:01.322879 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:01.327221 (+ 4342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.328259 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.328265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.328268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.328436 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.328943 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.328949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.328952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.329342 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:01.329512 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:01.330015 (+ 503us) server_negotiation.cc:299] Negotiation successful
0928 03:05:01.330248 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1338,"thread_start_us":1225,"threads_started":1}
I20260928 03:05:01.330699 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.296906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:51114)
0928 03:05:01.308419 (+ 11513us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:01.308475 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:01.308635 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:01.310092 (+ 1457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:01.310103 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:01.311227 (+ 1124us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:01.311785 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.311805 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.313982 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.313987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.316141 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.316158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.317438 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.317446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.317620 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.319050 (+ 1430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:01.319080 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:01.322530 (+ 3450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:01.327365 (+ 4835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:01.327374 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:01.327380 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:01.328080 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:01.328554 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:01.328558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:01.328561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:01.328764 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:01.329491 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:01.329499 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:01.329684 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 03:05:01.329884 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 03:05:01.330237 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11211,"thread_start_us":11026,"threads_started":1}
I20260928 03:05:01.332492 13788 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:05:01 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564701, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.60@KRBTEST.COM for oryx/127.6.192.61@KRBTEST.COM
I20260928 03:05:01.347147 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.333204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:41467 (local address 127.0.0.1:47096)
0928 03:05:01.333485 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:01.333502 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:01.333626 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:01.334094 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:01.334099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:01.334513 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:01.334664 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.334673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.335841 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.335850 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.336889 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.336901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.337722 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.337729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.337947 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.338611 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:01.338630 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:01.340978 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:01.344536 (+ 3558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:01.344541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:01.344546 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:01.345049 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:01.345471 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:01.345476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:01.345480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:01.345625 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:01.346141 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:01.346146 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:01.346303 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 03:05:01.346445 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 03:05:01.346782 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260928 03:05:01.347162 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.333516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47096 (local address 127.6.192.61:41467)
0928 03:05:01.333732 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:01.333738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:01.333812 (+ 74us) server_negotiation.cc:407] Connection header received
0928 03:05:01.333864 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:01.333868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:01.333939 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:01.334069 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:01.334844 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.335698 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.337045 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.337538 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.337702 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.341254 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:01.341260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:01.341264 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:01.341300 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:01.344376 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.345183 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.345190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.345198 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.345352 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.345733 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.345740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.345743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.346023 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:01.346176 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:01.346530 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 03:05:01.346780 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 03:05:01.348435 13788 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:05:01 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564701, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.60@KRBTEST.COM for oryx/127.6.192.60@KRBTEST.COM
I20260928 03:05:01.362509 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.349201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40419 (local address 127.0.0.1:60526)
0928 03:05:01.349554 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:01.349574 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:01.349675 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:01.350074 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:01.350077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:01.350453 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:01.350600 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.350608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.351711 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.351716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.352844 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.352857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.353586 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.353592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.353845 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.354525 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:01.354544 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:01.356873 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:01.360127 (+ 3254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:01.360130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:01.360133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:01.360560 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:01.360932 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:01.360936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:01.360938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:01.361085 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:01.361576 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:01.361579 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:01.361723 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 03:05:01.361887 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 03:05:01.362187 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260928 03:05:01.362778 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.349657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.6.192.60:40419)
0928 03:05:01.349839 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:01.349843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:01.349858 (+ 15us) server_negotiation.cc:407] Connection header received
0928 03:05:01.349898 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:01.349901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:01.349954 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:01.350036 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:01.350735 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.351419 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.352979 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.353395 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.353495 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.357112 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:01.357117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:01.357120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:01.357150 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:01.359984 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.360699 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.360703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.360705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.360822 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.361178 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.361182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.361184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.361472 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:01.361580 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:01.362005 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 03:05:01.362413 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":80}
I20260928 03:05:01.368105 13788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: Bootstrap starting.
I20260928 03:05:01.371354 13788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:01.372460 13788 log.cc:824] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: Log is configured to *not* fsync() on all Append() calls
I20260928 03:05:01.375342 13788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: No bootstrap required, opened a new log
I20260928 03:05:01.383039 13788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } }
I20260928 03:05:01.383530 13788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:01.383664 13788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04b9a59f6fd94c6892c5278980919e61, State: Initialized, Role: FOLLOWER
I20260928 03:05:01.384378 13788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } }
I20260928 03:05:01.385288 13798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } } }
I20260928 03:05:01.385627 13798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:05:01.386417 13788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 03:05:01.396286 13809 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 03:05:01.396435 13809 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 03:05:01.400633 13634 master_runner.cc:428] Detected that this master 04b9a59f6fd94c6892c5278980919e61 is joining an existing cluster
I20260928 03:05:01.400779 13634 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.192.60:40419
I20260928 03:05:01.426268 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.405600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51120 (local address 127.6.192.62:41507)
0928 03:05:01.405824 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:01.405832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:01.405984 (+ 152us) server_negotiation.cc:407] Connection header received
0928 03:05:01.406325 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:01.406333 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:01.406431 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:01.406506 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:01.408439 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.409956 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.413070 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.414066 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.414202 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.417988 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:01.417997 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:01.418004 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:01.418050 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:01.421867 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.422836 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.422843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.422847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.423031 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.423638 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.423643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.423646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.424034 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:01.424201 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:01.425514 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0928 03:05:01.425823 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 03:05:01.426440 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.405088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:51120)
0928 03:05:01.405868 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:01.405894 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:01.406105 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:01.406758 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:01.406763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:01.407690 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:01.408178 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.408196 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.410274 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.410283 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.412805 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.412828 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.414671 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.414680 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.414967 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.416242 (+ 1275us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:01.416272 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:01.417718 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:01.422077 (+ 4359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:01.422082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:01.422089 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:01.422646 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:01.423182 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:01.423187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:01.423190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:01.423457 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:01.424268 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:01.424274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:01.425107 (+ 833us) client_negotiation.cc:769] Sending connection context
0928 03:05:01.425370 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 03:05:01.425787 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"spinlock_wait_cycles":2944,"thread_start_us":907,"threads_started":1}
I20260928 03:05:01.435552 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.406490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:41467 (local address 127.0.0.1:47102)
0928 03:05:01.416494 (+ 10004us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:01.416516 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:01.416780 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:01.417320 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:01.417326 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:01.418312 (+ 986us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:01.418907 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.418920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.420511 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.420516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.422568 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.422584 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.423957 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.423963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.424198 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.425109 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:01.425139 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:01.427115 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:01.432002 (+ 4887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:01.432008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:01.432012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:01.432608 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:01.433112 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:01.433117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:01.433121 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:01.433319 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:01.434108 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:01.434113 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:01.434404 (+ 291us) client_negotiation.cc:769] Sending connection context
0928 03:05:01.434580 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 03:05:01.434920 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9865,"mutex_wait_us":262,"spinlock_wait_cycles":78208,"thread_start_us":9264,"threads_started":1}
I20260928 03:05:01.436477 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.407023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47102 (local address 127.6.192.61:41467)
0928 03:05:01.407231 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:01.407236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:01.416612 (+ 9376us) server_negotiation.cc:407] Connection header received
0928 03:05:01.416934 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:01.416939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:01.417027 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:01.417109 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:01.419093 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.420308 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.422752 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.423512 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.423630 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.427374 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:01.427381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:01.427385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:01.427431 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:01.431796 (+ 4365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.432787 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.432794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.432798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.432964 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.433468 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.433475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.433480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.433881 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:01.434031 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:01.435883 (+ 1852us) server_negotiation.cc:299] Negotiation successful
0928 03:05:01.436140 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 03:05:01.437916 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.405277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40419 (local address 127.0.0.1:60538)
0928 03:05:01.416494 (+ 11217us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:01.416516 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:01.416699 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:01.417263 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:01.417267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:01.418304 (+ 1037us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:01.418907 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.418920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.420524 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.420528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.422568 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.422584 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.424024 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.424033 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.424252 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.425109 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:01.425147 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:01.427556 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:01.433755 (+ 6199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:01.433760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:01.433763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:01.434276 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:01.434696 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:01.434700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:01.434703 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:01.434848 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:01.436318 (+ 1470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:01.436324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:01.436508 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 03:05:01.437086 (+ 578us) client_negotiation.cc:240] Negotiation successful
0928 03:05:01.437373 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11024,"mutex_wait_us":204,"spinlock_wait_cycles":1408,"thread_start_us":10463,"threads_started":1}
I20260928 03:05:01.438952 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.405807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.6.192.60:40419)
0928 03:05:01.406001 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:01.406007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:01.416591 (+ 10584us) server_negotiation.cc:407] Connection header received
0928 03:05:01.416866 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:01.416874 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:01.416942 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:01.417017 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:01.419065 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.420307 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.422793 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.423580 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.423660 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.427739 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:01.427746 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:01.427751 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:01.427801 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:01.433603 (+ 5802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.434430 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.434434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.434441 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.434594 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.434967 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.434972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.434977 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.435351 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:01.435561 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:01.438335 (+ 2774us) server_negotiation.cc:299] Negotiation successful
0928 03:05:01.438582 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 03:05:01.439299 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 13134
Sep 28 03:05:01 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564698, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.62@KRBTEST.COM for oryx/127.6.192.60@KRBTEST.COM
Sep 28 03:05:01 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.192.1@KRBTEST.COM
I20260928 03:05:01.456295 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.433283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40419 (local address 127.0.0.1:60546)
0928 03:05:01.433665 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:01.433693 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:01.433817 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:01.438721 (+ 4904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:01.438727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:01.439291 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:01.439543 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.439562 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.441624 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.441630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.442742 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.442761 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.443837 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.443844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.444099 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.444844 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:01.444869 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:01.447930 (+ 3061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:01.452945 (+ 5015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:01.452952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:01.452957 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:01.453586 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:01.454161 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:01.454169 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:01.454176 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:01.454383 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:01.455193 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:01.455199 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:01.455380 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 03:05:01.455605 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 03:05:01.455925 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253}
I20260928 03:05:01.456374 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.434050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.6.192.60:40419)
0928 03:05:01.438236 (+ 4186us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:01.438242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:01.438261 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:05:01.438370 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:01.438375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:01.438494 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:01.439648 (+ 1154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:01.440016 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.441405 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.442882 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.443629 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.444454 (+ 825us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.448224 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:01.448232 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:01.448237 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:01.448282 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:01.452682 (+ 4400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.453818 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.453825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.453828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.454004 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.454532 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.454537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.454540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.455038 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:01.455215 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:01.455700 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 03:05:01.455928 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4045,"thread_start_us":3857,"threads_started":1}
I20260928 03:05:01.458201 13073 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.192.60:40419: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "04b9a59f6fd94c6892c5278980919e61"
member_type: NON_VOTER
last_known_addr {
host: "127.6.192.60"
port: 40419
}
attrs {
promote: true
}
}
dest_uuid: "7a2716d3b7d545dd82f8fadedefd92d9"
cas_config_opid_index: 5
I20260928 03:05:01.459184 13073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: NON_VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: true } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: NON_VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: true } }
I20260928 03:05:01.461076 13792 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 04b9a59f6fd94c6892c5278980919e61 (127.6.192.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: NON_VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: true } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: NON_VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: true } } }
I20260928 03:05:01.461984 13682 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 1 LEARNER]: Refusing update from remote peer 7a2716d3b7d545dd82f8fadedefd92d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 03:05:01.464759 13763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:01.468894 13763 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 1 FOLLOWER]: Refusing update from remote peer 7a2716d3b7d545dd82f8fadedefd92d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 03:05:01.473858 13820 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: NON_VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 03:05:01.474843 13820 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: NON_VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:05:01.479647 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.441082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49484 (local address 127.6.192.1:45339)
0928 03:05:01.441932 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:01.441941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:01.441977 (+ 36us) server_negotiation.cc:407] Connection header received
0928 03:05:01.442054 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:01.442064 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:01.442364 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:01.442557 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:01.444135 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.445832 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.447795 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.448750 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.448899 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.454955 (+ 6056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:01.454981 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:01.455004 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:01.455051 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:01.460232 (+ 5181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.473833 (+ 13601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.473846 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.473850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.474078 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.475780 (+ 1702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.475787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.475792 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.476319 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:01.476650 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:01.478798 (+ 2148us) server_negotiation.cc:299] Negotiation successful
0928 03:05:01.479128 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":494,"threads_started":1}
I20260928 03:05:01.481521 13257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:01.485893 13792 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.192.60:40419
I20260928 03:05:01.486884 13634 master.cc:537] Master@127.6.192.60:40419 shutting down...
I20260928 03:05:01.490391 13821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: NON_VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: true } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: NON_VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: true } } }
I20260928 03:05:01.490854 13821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: This master's current role is: LEADER
W20260928 03:05:01.493884 13057 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 -> Peer 04b9a59f6fd94c6892c5278980919e61 (127.6.192.60:40419): Couldn't send request to peer 04b9a59f6fd94c6892c5278980919e61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 03:05:01.493372 13682 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 04b9a59f6fd94c6892c5278980919e61 (127.6.192.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: NON_VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: true } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: NON_VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: true } } }
I20260928 03:05:01.502749 13710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: NON_VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: true } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: NON_VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: true } } }
I20260928 03:05:01.503091 13710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [sys.catalog]: This master's current role is: LEARNER
I20260928 03:05:01.504202 13820 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9: attempting to promote NON_VOTER 569125e0394b44a8a3d58d5fb58d6821 to VOTER
I20260928 03:05:01.505232 13820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: NON_VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: true } }
I20260928 03:05:01.508255 13682 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 1 LEARNER]: Refusing update from remote peer 7a2716d3b7d545dd82f8fadedefd92d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260928 03:05:01.508597 13057 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 -> Peer 04b9a59f6fd94c6892c5278980919e61 (127.6.192.60:40419): Couldn't send request to peer 04b9a59f6fd94c6892c5278980919e61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 03:05:01.508996 13792 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 03:05:01.512020 13821 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 569125e0394b44a8a3d58d5fb58d6821 (127.6.192.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: NON_VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: true } } }
I20260928 03:05:01.515970 13682 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 569125e0394b44a8a3d58d5fb58d6821 (127.6.192.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: NON_VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: true } } }
I20260928 03:05:01.517390 13820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: NON_VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: true } } }
I20260928 03:05:01.517694 13820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: This master's current role is: LEADER
I20260928 03:05:01.518127 13820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: NON_VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: true } } }
I20260928 03:05:01.518340 13820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: This master's current role is: LEADER
W20260928 03:05:01.520570 13742 tablet.cc:2369] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:05:01.520812 13710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: NON_VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: true } } }
I20260928 03:05:01.521062 13710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:05:01.523332 13634 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:05:01.524926 13634 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:01.525254 13634 tablet_replica.cc:330] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: stopping tablet replica
I20260928 03:05:01.532858 13634 master.cc:552] Master@127.6.192.60:40419 shutdown complete.
I20260928 03:05:01.532966 13634 master_runner.cc:305] Clearing existing system tablet
I20260928 03:05:01.533378 13634 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:01.533480 13634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:01.536676 13634 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 03:05:01.538499 13828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:01.538968 13634 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:01.539237 13634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 03:05:01.539346 13634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/wal
uuid: "04b9a59f6fd94c6892c5278980919e61"
format_stamp: "Formatted at 2026-09-28 03:05:01 on dist-test-slave-fjww"
server_key: "64fbe2376e5ebffdfaef9d5d67cf31df"
server_key_iv: "cba08deba4e9bb9f376fd7f3ea593dac"
server_key_version: "encryptionkey@0"
I20260928 03:05:01.544731 13634 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:01.552219 13634 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 03:05:01.552424 13634 log.cc:1196] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 03:05:01.552918 13634 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: Deleting consensus metadata
I20260928 03:05:01.553195 13634 master_runner.cc:315] Copying system tablet from 127.6.192.62:41507
I20260928 03:05:01.561362 13634 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: tablet copy: Beginning tablet copy session from remote peer at address 127.6.192.62:41507
I20260928 03:05:01.583243 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.562376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51132 (local address 127.6.192.62:41507)
0928 03:05:01.562620 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:01.562627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:01.568188 (+ 5561us) server_negotiation.cc:407] Connection header received
0928 03:05:01.568400 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:01.568406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:01.568496 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:01.568615 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:01.569854 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.571023 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.573057 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.573772 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.573918 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.576429 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:01.576435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:01.576440 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:01.576479 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:01.580005 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.580919 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.580925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.580930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.581095 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:01.581570 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:01.581576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:01.581579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:01.581944 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:01.582104 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:01.582640 (+ 536us) server_negotiation.cc:299] Negotiation successful
0928 03:05:01.582887 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260928 03:05:01.583249 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.562342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:51132)
0928 03:05:01.568008 (+ 5666us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:01.568032 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:01.568225 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:01.568676 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:01.568681 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:01.569320 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:01.569618 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.569635 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.571195 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.571201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.572854 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.572872 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.573962 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.573969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.574155 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.574932 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:01.574959 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:01.576182 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:01.580173 (+ 3991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:01.580180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:01.580185 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:01.580752 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:01.581222 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:01.581228 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:01.581232 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:01.581431 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:01.582081 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:01.582087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:01.582318 (+ 231us) client_negotiation.cc:769] Sending connection context
0928 03:05:01.582470 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 03:05:01.582824 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5483,"thread_start_us":5310,"threads_started":1}
I20260928 03:05:01.584344 13093 tablet_copy_service.cc:140] P 7a2716d3b7d545dd82f8fadedefd92d9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 04b9a59f6fd94c6892c5278980919e61 ({username='oryx', principal='oryx/127.6.192.60@KRBTEST.COM'} at 127.0.0.1:51132)
I20260928 03:05:01.584527 13093 tablet_copy_service.cc:161] P 7a2716d3b7d545dd82f8fadedefd92d9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 04b9a59f6fd94c6892c5278980919e61 at {username='oryx', principal='oryx/127.6.192.60@KRBTEST.COM'} at 127.0.0.1:51132: session id = 04b9a59f6fd94c6892c5278980919e61-00000000000000000000000000000000
I20260928 03:05:01.588171 13093 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 03:05:01.590199 13634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 03:05:01.597610 13634 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: tablet copy: Starting download of 0 data blocks...
I20260928 03:05:01.597957 13634 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: tablet copy: Starting download of 1 WAL segments...
I20260928 03:05:01.600785 13634 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 03:05:01.607239 13093 tablet_copy_service.cc:342] P 7a2716d3b7d545dd82f8fadedefd92d9: Request end of tablet copy session 04b9a59f6fd94c6892c5278980919e61-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.192.60@KRBTEST.COM'} at 127.0.0.1:51132
I20260928 03:05:01.607432 13093 tablet_copy_service.cc:433] P 7a2716d3b7d545dd82f8fadedefd92d9: ending tablet copy session 04b9a59f6fd94c6892c5278980919e61-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 04b9a59f6fd94c6892c5278980919e61
I20260928 03:05:01.607604 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 13134
I20260928 03:05:01.621163 13634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:01.624187 13835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:01.627719 13836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:01.633087 13838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:01.633302 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:45339
--local_ip_for_outbound_sockets=127.6.192.1
--tserver_master_addrs=127.6.192.62:41507
--webserver_port=41905
--webserver_interface=127.6.192.1
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.1.keytab
--principal=oryx/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.192.62:41507,127.6.192.61:41467
--tserver_master_addrs=127.6.192.62:41507,127.6.192.61:41467,127.6.192.60:40419 with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:05:01.635819 13634 server_base.cc:1034] running on GCE node
I20260928 03:05:01.636449 13634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:01.639770 13634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:01.641058 13634 hybrid_clock.cc:648] HybridClock initialized: now 1790564701640995 us; error 54 us; skew 500 ppm
I20260928 03:05:01.641335 13634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:01 dist-test-slave-fjww krb5kdc[13028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564701, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:01.643729 13634 init.cc:377] Logged in from keytab as oryx/127.6.192.60@KRBTEST.COM (short username oryx)
I20260928 03:05:01.645275 13634 webserver.cc:460] Webserver started at http://127.6.192.60:42543/ using document root <none> and password file <none>
I20260928 03:05:01.645650 13634 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:01.645751 13634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:01.649144 13634 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 03:05:01.651314 13846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:01.651796 13634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 03:05:01.651952 13634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/wal
uuid: "04b9a59f6fd94c6892c5278980919e61"
format_stamp: "Formatted at 2026-09-28 03:05:01 on dist-test-slave-fjww"
server_key: "64fbe2376e5ebffdfaef9d5d67cf31df"
server_key_iv: "cba08deba4e9bb9f376fd7f3ea593dac"
server_key_version: "encryptionkey@0"
I20260928 03:05:01.652161 13634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:01.700795 13634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:01.705926 13634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:01.746316 13634 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.60:40419
I20260928 03:05:01.746377 13897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.60:40419 every 8 connection(s)
I20260928 03:05:01.747408 13634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/master-2/data/info.pb
I20260928 03:05:01.753257 13898 sys_catalog.cc:263] Verifying existing consensus state
I20260928 03:05:01.757701 13898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: Bootstrap starting.
I20260928 03:05:01.778635 13898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 03:05:01.779220 13898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: Bootstrap complete.
I20260928 03:05:01.780392 13898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: NON_VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: true } }
I20260928 03:05:01.780696 13898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 04b9a59f6fd94c6892c5278980919e61, State: Initialized, Role: LEARNER
I20260928 03:05:01.780987 13898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: NON_VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: true } }
I20260928 03:05:01.782253 13898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:05:01.781934 13901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: NON_VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: true } } }
I20260928 03:05:01.782528 13901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [sys.catalog]: This master's current role is: LEARNER
I20260928 03:05:01.792286 13912 catalog_manager.cc:1259] Loaded cluster ID: 3459c6c9524d4c0eab848e651e9c2e9c
I20260928 03:05:01.792418 13912 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: loading cluster ID for follower catalog manager: success
I20260928 03:05:01.796556 13912 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: acquiring CA information for follower catalog manager: success
I20260928 03:05:01.799171 13912 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 03:05:01.944332 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.936855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40419 (local address 127.0.0.1:60548)
0928 03:05:01.937134 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:01.937159 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:01.937367 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:01.938152 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:01.938160 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:01.938198 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:01.938424 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.938438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.939867 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.939873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.941837 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:01.941862 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.943209 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.943216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:01.943486 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.943533 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 03:05:01.943648 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 03:05:01.943911 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 03:05:01.944461 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:01.937156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60548 (local address 127.6.192.60:40419)
0928 03:05:01.937737 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:01.937741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:01.937758 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:05:01.937849 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:01.937854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:01.937951 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:01.938100 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:01.938600 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.939712 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.942031 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:01.942913 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:01.943043 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:01.943264 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:01.943813 (+ 549us) server_negotiation.cc:299] Negotiation successful
0928 03:05:01.943977 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":344,"threads_started":1}
I20260928 03:05:01.945619 13873 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 03:05:01.946334 13901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a2716d3b7d545dd82f8fadedefd92d9. Latest consensus state: current_term: 1 leader_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: NON_VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: true } } }
I20260928 03:05:01.946676 13901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [sys.catalog]: This master's current role is: LEARNER
W20260928 03:05:02.044920 13841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:02.045222 13841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:02.045382 13841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:02.053400 13841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:02.053531 13841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:02.053719 13841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:05:02.063858 13841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.1.keytab
--principal=oryx/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:45339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=41905
--tserver_master_addrs=127.6.192.62:41507,127.6.192.61:41467,127.6.192.60:40419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:02.065280 13841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:02.066773 13841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:02.082609 13919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:02.082707 13918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:02.082569 13921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:02.083542 13841 server_base.cc:1034] running on GCE node
I20260928 03:05:02.084342 13841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:02.085393 13841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:02.086647 13841 hybrid_clock.cc:648] HybridClock initialized: now 1790564702086553 us; error 60 us; skew 500 ppm
I20260928 03:05:02.087045 13841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:02 dist-test-slave-fjww krb5kdc[13028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564702, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:02.090691 13841 init.cc:377] Logged in from keytab as oryx/127.6.192.1@KRBTEST.COM (short username oryx)
I20260928 03:05:02.092391 13841 webserver.cc:460] Webserver started at http://127.6.192.1:41905/ using document root <none> and password file <none>
I20260928 03:05:02.093132 13841 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:02.093250 13841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:02.099239 13841 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 03:05:02.102250 13928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:02.104110 13841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:05:02.104341 13841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "b258d7d74c2941b983bb7bfe5d709c7d"
format_stamp: "Formatted at 2026-09-28 03:04:58 on dist-test-slave-fjww"
server_key: "d50b2858fa7476d0fb01e971ffacecc9"
server_key_iv: "a4c73ad91e2f1b76347a73a8fb69e528"
server_key_version: "encryptionkey@0"
I20260928 03:05:02.105567 13841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:02.125523 13841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:02.131376 13841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:02.131767 13841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:02.132963 13841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:05:02.134528 13841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:02.134631 13841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:02.134723 13841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:02.134786 13841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:02.170977 13841 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:45339
I20260928 03:05:02.171038 14040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:45339 every 8 connection(s)
I20260928 03:05:02.172623 13841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:05:02.173408 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 13841
I20260928 03:05:02.173786 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 13268
Sep 28 03:05:02 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.192.2@KRBTEST.COM
Sep 28 03:05:02 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564702, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.60@KRBTEST.COM
Sep 28 03:05:02 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564702, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM
I20260928 03:05:02.207690 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:02.175929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:42427 (local address 127.6.192.60:40419)
0928 03:05:02.177533 (+ 1604us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:02.177543 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:02.184319 (+ 6776us) server_negotiation.cc:407] Connection header received
0928 03:05:02.186636 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:02.186643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:02.186729 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:02.186796 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:02.188288 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.189356 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.191888 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.192674 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.192783 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:02.199100 (+ 6317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:02.199109 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:02.199114 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:02.199166 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:02.202936 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:02.204123 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:02.204129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:02.204132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:02.204300 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:02.204928 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:02.204933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:02.204936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:02.205308 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:02.205422 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:02.207041 (+ 1619us) server_negotiation.cc:299] Negotiation successful
0928 03:05:02.207306 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1421}
I20260928 03:05:02.209440 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:02.175260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53116 (local address 127.6.192.2:36405)
0928 03:05:02.176599 (+ 1339us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:02.176612 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:02.177221 (+ 609us) server_negotiation.cc:407] Connection header received
0928 03:05:02.177458 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:02.177470 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:02.177766 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:02.178014 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:02.180231 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.181914 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.184457 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.185339 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.185485 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:02.199791 (+ 14306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:02.199814 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:02.199854 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:02.199934 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:02.204710 (+ 4776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:02.205777 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:02.205788 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:02.205791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:02.205957 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:02.207481 (+ 1524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:02.207489 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:02.207494 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:02.207973 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:02.208269 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:02.208655 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 03:05:02.208970 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1093,"thread_start_us":936,"threads_started":1}
I20260928 03:05:02.209887 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:02.176690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.1:38667)
0928 03:05:02.179257 (+ 2567us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:02.179317 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:02.182309 (+ 2992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:02.183574 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:02.183588 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:02.184528 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:02.185431 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:02.185462 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.187366 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.187374 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:02.189828 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:02.189842 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.195932 (+ 6090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.195939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:02.196136 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:02.197862 (+ 1726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:02.197890 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:02.200888 (+ 2998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:02.205475 (+ 4587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:02.205478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:02.205481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:02.205966 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:02.206425 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:02.206429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:02.206432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:02.206599 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:02.207952 (+ 1353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:02.207957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:02.208415 (+ 458us) client_negotiation.cc:769] Sending connection context
0928 03:05:02.208559 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 03:05:02.208857 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2168,"thread_start_us":7020,"threads_started":1}
I20260928 03:05:02.209935 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:02.178165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40419 (local address 127.6.192.1:42427)
0928 03:05:02.184209 (+ 6044us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:02.184229 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:02.186459 (+ 2230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:02.186997 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:02.187002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:02.187797 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:02.188107 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:02.188119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.189565 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.189575 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:02.191696 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:02.191715 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.193176 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.193186 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:02.193566 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:02.194463 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:02.194507 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:02.198758 (+ 4251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:02.203148 (+ 4390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:02.203164 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:02.203189 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:02.203965 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:02.204460 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:02.204464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:02.204468 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:02.204774 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:02.206157 (+ 1383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:02.206163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:02.206573 (+ 410us) client_negotiation.cc:769] Sending connection context
0928 03:05:02.206935 (+ 362us) client_negotiation.cc:240] Negotiation successful
0928 03:05:02.207569 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5901,"thread_start_us":548,"threads_started":1}
I20260928 03:05:02.211086 13391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:02.211671 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:02.177721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:38667 (local address 127.6.192.62:41507)
0928 03:05:02.178545 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:02.178552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:02.179464 (+ 912us) server_negotiation.cc:407] Connection header received
0928 03:05:02.182675 (+ 3211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:02.182681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:02.182780 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:02.182864 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:02.185716 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.187137 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.194674 (+ 7537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.195414 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.195554 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:02.201141 (+ 5587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:02.201147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:02.201150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:02.201212 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:02.204909 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:02.206122 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:02.206128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:02.206131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:02.206305 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:02.206715 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:02.206720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:02.206724 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:02.207097 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:02.207257 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:02.210907 (+ 3650us) server_negotiation.cc:299] Negotiation successful
0928 03:05:02.211223 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":637,"threads_started":1}
Sep 28 03:05:02 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564702, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.61@KRBTEST.COM
I20260928 03:05:02.212677 14041 heartbeater.cc:347] Connected to a master server at 127.6.192.60:40419
I20260928 03:05:02.212849 14042 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41507
I20260928 03:05:02.213064 14042 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:02.213071 14041 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:02.214394 14042 heartbeater.cc:511] Master 127.6.192.62:41507 requested a full tablet report, sending...
I20260928 03:05:02.214438 14041 heartbeater.cc:511] Master 127.6.192.60:40419 requested a full tablet report, sending...
I20260928 03:05:02.216403 13073 ts_manager.cc:194] Re-registered known tserver with Master: b258d7d74c2941b983bb7bfe5d709c7d (127.6.192.1:45339)
I20260928 03:05:02.217268 13863 ts_manager.cc:194] Registered new tserver with Master: b258d7d74c2941b983bb7bfe5d709c7d (127.6.192.1:45339)
I20260928 03:05:02.217760 13073 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:38667
I20260928 03:05:02.222311 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:02.188492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:41467 (local address 127.6.192.1:34555)
0928 03:05:02.196884 (+ 8392us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:02.196904 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:02.197031 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:02.201742 (+ 4711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:02.201750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:02.202357 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:02.202850 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:02.202866 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.204578 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.204585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:02.206110 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:02.206125 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.207356 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.207365 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:02.207619 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:02.209309 (+ 1690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:02.209331 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:02.214008 (+ 4677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:02.219418 (+ 5410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:02.219423 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:02.219427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:02.219969 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:02.220530 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:02.220534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:02.220536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:02.220699 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:02.221447 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:02.221451 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:02.221577 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 03:05:02.221696 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 03:05:02.222006 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8230,"thread_start_us":5637,"threads_started":1}
I20260928 03:05:02.222733 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:02.184724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:34555 (local address 127.6.192.61:41467)
0928 03:05:02.201283 (+ 16559us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:02.201292 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:02.201327 (+ 35us) server_negotiation.cc:407] Connection header received
0928 03:05:02.201398 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:02.201407 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:02.201527 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:02.203027 (+ 1500us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:02.203243 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.204373 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.206313 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.206959 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.207084 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:02.214334 (+ 7250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:02.214343 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:02.214351 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:02.214404 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:02.219247 (+ 4843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:02.220127 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:02.220135 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:02.220140 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:02.220368 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:02.220839 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:02.220846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:02.220851 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:02.221258 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:02.221427 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:02.221875 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 03:05:02.222160 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16381,"thread_start_us":15332,"threads_started":1}
I20260928 03:05:02.223753 14043 heartbeater.cc:347] Connected to a master server at 127.6.192.61:41467
I20260928 03:05:02.224010 14043 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:02.224525 14043 heartbeater.cc:511] Master 127.6.192.61:41467 requested a full tablet report, sending...
I20260928 03:05:02.226388 13672 ts_manager.cc:194] Registered new tserver with Master: b258d7d74c2941b983bb7bfe5d709c7d (127.6.192.1:45339)
I20260928 03:05:02.301960 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 13268
I20260928 03:05:02.319094 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:36405
--local_ip_for_outbound_sockets=127.6.192.2
--tserver_master_addrs=127.6.192.62:41507
--webserver_port=38199
--webserver_interface=127.6.192.2
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.2.keytab
--principal=oryx/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.192.62:41507,127.6.192.61:41467
--tserver_master_addrs=127.6.192.62:41507,127.6.192.61:41467,127.6.192.60:40419 with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:05:02.328227 13821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: NON_VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: true } } }
I20260928 03:05:02.328599 13821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: This master's current role is: LEADER
I20260928 03:05:02.330111 14055 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9: attempting to promote NON_VOTER 04b9a59f6fd94c6892c5278980919e61 to VOTER
I20260928 03:05:02.330878 14055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } }
I20260928 03:05:02.332908 13873 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 1 LEARNER]: Refusing update from remote peer 7a2716d3b7d545dd82f8fadedefd92d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 03:05:02.332931 13682 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 1 FOLLOWER]: Refusing update from remote peer 7a2716d3b7d545dd82f8fadedefd92d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 03:05:02.333652 14054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 03:05:02.334167 13821 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 03:05:02.336705 13821 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 04b9a59f6fd94c6892c5278980919e61 (127.6.192.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } } }
I20260928 03:05:02.340234 13682 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 04b9a59f6fd94c6892c5278980919e61 (127.6.192.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } } }
I20260928 03:05:02.339975 13873 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 04b9a59f6fd94c6892c5278980919e61 (127.6.192.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } } }
I20260928 03:05:02.344250 14055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } } }
I20260928 03:05:02.344301 14054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } } }
I20260928 03:05:02.344576 14055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: This master's current role is: LEADER
I20260928 03:05:02.344658 14054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: This master's current role is: LEADER
I20260928 03:05:02.345034 14054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } } }
I20260928 03:05:02.345276 14054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9 [sys.catalog]: This master's current role is: LEADER
I20260928 03:05:02.345521 13901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } } }
I20260928 03:05:02.345861 13901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:05:02.346556 14058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } } }
I20260928 03:05:02.346920 14058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 03:05:02.673403 14053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:02.673897 14053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:02.674067 14053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:02.681941 14053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:02.682125 14053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:02.682322 14053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:05:02.692823 14053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.2.keytab
--principal=oryx/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:36405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=38199
--tserver_master_addrs=127.6.192.62:41507,127.6.192.61:41467,127.6.192.60:40419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:02.694285 14053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:02.695768 14053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:02.710974 14064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:02.711018 14063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:02.711678 14053 server_base.cc:1034] running on GCE node
W20260928 03:05:02.711704 14066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:02.712781 14053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:02.713814 14053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:02.715085 14053 hybrid_clock.cc:648] HybridClock initialized: now 1790564702714994 us; error 78 us; skew 500 ppm
I20260928 03:05:02.715502 14053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:02 dist-test-slave-fjww krb5kdc[13028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564702, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:02.719604 14053 init.cc:377] Logged in from keytab as oryx/127.6.192.2@KRBTEST.COM (short username oryx)
I20260928 03:05:02.721640 14053 webserver.cc:460] Webserver started at http://127.6.192.2:38199/ using document root <none> and password file <none>
I20260928 03:05:02.722452 14053 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:02.722590 14053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:02.729636 14053 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 03:05:02.733114 14073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:02.734822 14053 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:05:02.735047 14053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "b73e7c5e06f7404c928ec74316dd4512"
format_stamp: "Formatted at 2026-09-28 03:04:59 on dist-test-slave-fjww"
server_key: "96e20727adf81484800233cb5aeb4c7e"
server_key_iv: "c7153f52d641ec549fe2eca357bada94"
server_key_version: "encryptionkey@0"
I20260928 03:05:02.736224 14053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:02.766744 14053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:02.771660 14053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:02.771917 14053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:02.773108 14053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:05:02.775091 14053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:02.775200 14053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:02.775393 14053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:02.775482 14053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:02.819819 14053 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:36405
I20260928 03:05:02.819880 14185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:36405 every 8 connection(s)
I20260928 03:05:02.821696 14053 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:05:02.825699 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 14053
I20260928 03:05:02.826122 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 13402
Sep 28 03:05:02 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.192.3@KRBTEST.COM
Sep 28 03:05:02 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564702, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.60@KRBTEST.COM
Sep 28 03:05:02 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564702, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM
Sep 28 03:05:02 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564702, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.61@KRBTEST.COM
I20260928 03:05:02.865540 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:02.827155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45108 (local address 127.6.192.3:44019)
0928 03:05:02.837378 (+ 10223us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:02.837393 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:02.840171 (+ 2778us) server_negotiation.cc:407] Connection header received
0928 03:05:02.840593 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:02.840607 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:02.840978 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:02.841158 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:02.845497 (+ 4339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.847481 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.849464 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.850071 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.850228 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:02.856704 (+ 6476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:02.856725 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:02.856750 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:02.856814 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:02.861431 (+ 4617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:02.862485 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:02.862497 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:02.862501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:02.862682 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:02.863637 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:02.863643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:02.863647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:02.864032 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:02.864378 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:02.864758 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 03:05:02.865049 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9966,"thread_start_us":10244,"threads_started":1}
I20260928 03:05:02.867493 13525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:02.871254 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:02.841236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:41281 (local address 127.6.192.62:41507)
0928 03:05:02.846141 (+ 4905us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:02.846148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:02.846722 (+ 574us) server_negotiation.cc:407] Connection header received
0928 03:05:02.846852 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:02.846858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:02.846953 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:02.847127 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:02.850438 (+ 3311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.851776 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.855124 (+ 3348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.856040 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.856157 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:02.862633 (+ 6476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:02.862640 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:02.862645 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:02.862693 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:02.866517 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:02.867750 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:02.867758 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:02.867765 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:02.867938 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:02.868535 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:02.868541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:02.868545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:02.868929 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:02.869067 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:02.870564 (+ 1497us) server_negotiation.cc:299] Negotiation successful
0928 03:05:02.870839 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4747,"thread_start_us":4630,"threads_started":1}
I20260928 03:05:02.871569 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:02.825158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:56321 (local address 127.6.192.60:40419)
0928 03:05:02.844249 (+ 19091us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:02.844259 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:02.846057 (+ 1798us) server_negotiation.cc:407] Connection header received
0928 03:05:02.848952 (+ 2895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:02.848958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:02.849047 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:02.849166 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:02.850513 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.851969 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.855118 (+ 3149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.855916 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.856034 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:02.862270 (+ 6236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:02.862278 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:02.862282 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:02.862343 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:02.868228 (+ 5885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:02.869161 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:02.869167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:02.869170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:02.869329 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:02.869864 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:02.869869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:02.869872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:02.870227 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:02.870366 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:02.870894 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 03:05:02.871149 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18882,"thread_start_us":18657,"threads_started":1}
I20260928 03:05:02.871989 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:02.825555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:55395 (local address 127.6.192.61:41467)
0928 03:05:02.838084 (+ 12529us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:02.838094 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:02.838129 (+ 35us) server_negotiation.cc:407] Connection header received
0928 03:05:02.846884 (+ 8755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:02.846889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:02.846990 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:02.847127 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:02.851961 (+ 4834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.853300 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.855662 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.856355 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.856471 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:02.863356 (+ 6885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:02.863367 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:02.863371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:02.863434 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:02.867729 (+ 4295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:02.869022 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:02.869032 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:02.869038 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:02.869246 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:02.869906 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:02.869915 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:02.869921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:02.870430 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:02.870603 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:02.871323 (+ 720us) server_negotiation.cc:299] Negotiation successful
0928 03:05:02.871610 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12366,"thread_start_us":12258,"threads_started":1}
I20260928 03:05:02.873958 14196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:02.841144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.2:41281)
0928 03:05:02.846548 (+ 5404us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:02.846566 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:02.846711 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:02.847725 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:02.847733 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:02.848757 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:02.850133 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:02.850173 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.851985 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.851993 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:02.854904 (+ 2911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:02.854921 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.856492 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.856497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:02.856893 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:02.858047 (+ 1154us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:02.858092 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:02.862282 (+ 4190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:02.866740 (+ 4458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:02.866753 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:02.866773 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:02.867580 (+ 807us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:02.868110 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:02.868114 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:02.868118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:02.868401 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:02.869168 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:02.869176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:02.870079 (+ 903us) client_negotiation.cc:769] Sending connection context
0928 03:05:02.871604 (+ 1525us) client_negotiation.cc:240] Negotiation successful
0928 03:05:02.872271 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5278,"mutex_wait_us":591,"spinlock_wait_cycles":13568,"thread_start_us":5120,"threads_started":1}
I20260928 03:05:02.874011 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:02.825720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40419 (local address 127.6.192.2:56321)
0928 03:05:02.845931 (+ 20211us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:02.845958 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:02.848754 (+ 2796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:02.849220 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:02.849226 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:02.849936 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:02.850324 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:02.850336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.852316 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.852321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:02.854904 (+ 2583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:02.854921 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.856460 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.856475 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:02.856891 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:02.858041 (+ 1150us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:02.858089 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:02.861896 (+ 3807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:02.868396 (+ 6500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:02.868400 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:02.868403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:02.868990 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:02.869516 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:02.869520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:02.869524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:02.869699 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:02.870460 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:02.870465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:02.870598 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 03:05:02.871809 (+ 1211us) client_negotiation.cc:240] Negotiation successful
0928 03:05:02.872765 (+ 956us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19966,"mutex_wait_us":1007,"spinlock_wait_cycles":118656,"thread_start_us":19553,"threads_started":1}
I20260928 03:05:02.874099 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:02.825729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:41467 (local address 127.6.192.2:55395)
0928 03:05:02.834951 (+ 9222us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:02.835037 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:02.846466 (+ 11429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:02.847721 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:02.847731 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:02.848768 (+ 1037us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:02.851731 (+ 2963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:02.851754 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.853659 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.853665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:02.855480 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:02.855496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:02.856803 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:02.856810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:02.857031 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:02.858446 (+ 1415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:02.858474 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:02.863047 (+ 4573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:02.867976 (+ 4929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:02.867985 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:02.867992 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:02.868719 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:02.869468 (+ 749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:02.869476 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:02.869482 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:02.869746 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:02.870712 (+ 966us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:02.870720 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:02.870961 (+ 241us) client_negotiation.cc:769] Sending connection context
0928 03:05:02.872222 (+ 1261us) client_negotiation.cc:240] Negotiation successful
0928 03:05:02.872620 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8683,"mutex_wait_us":53,"spinlock_wait_cycles":62208,"thread_start_us":13598,"threads_started":1}
I20260928 03:05:02.875797 14188 heartbeater.cc:347] Connected to a master server at 127.6.192.61:41467
I20260928 03:05:02.875895 14187 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41507
I20260928 03:05:02.876164 14188 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:02.877105 14188 heartbeater.cc:511] Master 127.6.192.61:41467 requested a full tablet report, sending...
I20260928 03:05:02.877866 14186 heartbeater.cc:347] Connected to a master server at 127.6.192.60:40419
I20260928 03:05:02.878000 14186 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:02.878422 14186 heartbeater.cc:511] Master 127.6.192.60:40419 requested a full tablet report, sending...
I20260928 03:05:02.876168 14187 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:02.879009 13672 ts_manager.cc:194] Registered new tserver with Master: b73e7c5e06f7404c928ec74316dd4512 (127.6.192.2:36405)
I20260928 03:05:02.879130 14187 heartbeater.cc:511] Master 127.6.192.62:41507 requested a full tablet report, sending...
I20260928 03:05:02.879940 13863 ts_manager.cc:194] Registered new tserver with Master: b73e7c5e06f7404c928ec74316dd4512 (127.6.192.2:36405)
I20260928 03:05:02.880568 13073 ts_manager.cc:194] Re-registered known tserver with Master: b73e7c5e06f7404c928ec74316dd4512 (127.6.192.2:36405)
I20260928 03:05:02.881783 13073 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:41281
I20260928 03:05:02.967902 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 13402
I20260928 03:05:02.985139 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:44019
--local_ip_for_outbound_sockets=127.6.192.3
--tserver_master_addrs=127.6.192.62:41507
--webserver_port=45681
--webserver_interface=127.6.192.3
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.3.keytab
--principal=oryx/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.192.62:41507,127.6.192.61:41467
--tserver_master_addrs=127.6.192.62:41507,127.6.192.61:41467,127.6.192.60:40419 with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:05:03.220374 14042 heartbeater.cc:503] Master 127.6.192.62:41507 was elected leader, sending a full tablet report...
W20260928 03:05:03.359704 14199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:03.360044 14199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:03.360402 14199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:03.369154 14199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:03.369282 14199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:03.369482 14199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:05:03.380209 14199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.3.keytab
--principal=oryx/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:44019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=45681
--tserver_master_addrs=127.6.192.62:41507,127.6.192.61:41467,127.6.192.60:40419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:03.381634 14199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:03.383144 14199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:03.397948 14207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:03.397980 14205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:03.397948 14204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:03.399739 14199 server_base.cc:1034] running on GCE node
I20260928 03:05:03.400550 14199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:03.401551 14199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:03.402770 14199 hybrid_clock.cc:648] HybridClock initialized: now 1790564703402697 us; error 64 us; skew 500 ppm
I20260928 03:05:03.403072 14199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:03 dist-test-slave-fjww krb5kdc[13028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564703, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:03.406827 14199 init.cc:377] Logged in from keytab as oryx/127.6.192.3@KRBTEST.COM (short username oryx)
I20260928 03:05:03.408504 14199 webserver.cc:460] Webserver started at http://127.6.192.3:45681/ using document root <none> and password file <none>
I20260928 03:05:03.409235 14199 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:03.409382 14199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:03.415537 14199 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:05:03.418535 14214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:03.420399 14199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 03:05:03.420601 14199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "4af241c0395a4b3aa8178f812e35329c"
format_stamp: "Formatted at 2026-09-28 03:04:59 on dist-test-slave-fjww"
server_key: "5d0b2fd30dc6231a77dc7fc83317e40c"
server_key_iv: "7eaba3fe5a0264cc14aede74bc6064b6"
server_key_version: "encryptionkey@0"
I20260928 03:05:03.421692 14199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:03.453679 14199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:03.458978 14199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:03.459328 14199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:03.460637 14199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:05:03.462371 14199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:03.462471 14199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:03.462617 14199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:03.462728 14199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:03.499961 14199 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:44019
I20260928 03:05:03.500012 14326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:44019 every 8 connection(s)
I20260928 03:05:03.501613 14199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:05:03.509963 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 14199
Sep 28 03:05:03 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564703, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM
Sep 28 03:05:03 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564703, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.61@KRBTEST.COM
Sep 28 03:05:03 dist-test-slave-fjww krb5kdc[13028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564703, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.60@KRBTEST.COM
I20260928 03:05:03.570313 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.524763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:59241 (local address 127.6.192.62:41507)
0928 03:05:03.525633 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:03.525641 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:03.525663 (+ 22us) server_negotiation.cc:407] Connection header received
0928 03:05:03.526360 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:03.526369 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:03.526456 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:03.526562 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:03.536921 (+ 10359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.538559 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.544387 (+ 5828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.545508 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.545706 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.557581 (+ 11875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:03.557593 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:03.557603 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:03.557668 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:03.565723 (+ 8055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:03.567053 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:03.567061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:03.567064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:03.567239 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:03.568125 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:03.568131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:03.568134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:03.568535 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:03.568681 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:03.569680 (+ 999us) server_negotiation.cc:299] Negotiation successful
0928 03:05:03.569913 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":632,"threads_started":1}
I20260928 03:05:03.571647 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.524659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.3:59241)
0928 03:05:03.525426 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:03.525451 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:03.525849 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:03.527114 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:03.527130 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:03.536184 (+ 9054us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:03.536676 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:03.536693 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.538746 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.538756 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:03.544142 (+ 5386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:03.544174 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.545762 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.545773 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:03.546044 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.546935 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:03.546969 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:03.555278 (+ 8309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:03.565870 (+ 10592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:03.565878 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:03.565885 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:03.566902 (+ 1017us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:03.567723 (+ 821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:03.567727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:03.567731 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:03.568007 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:03.568679 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:03.568688 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:03.569180 (+ 492us) client_negotiation.cc:769] Sending connection context
0928 03:05:03.569565 (+ 385us) client_negotiation.cc:240] Negotiation successful
0928 03:05:03.570139 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":325,"threads_started":1}
I20260928 03:05:03.573467 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.505428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:41467 (local address 127.6.192.3:58295)
0928 03:05:03.524553 (+ 19125us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:03.524574 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:03.525770 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:03.527114 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:03.527130 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:03.529665 (+ 2535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:03.530977 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:03.531018 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.534531 (+ 3513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.534542 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:03.540176 (+ 5634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:03.540203 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.541900 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.541912 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:03.542366 (+ 454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.546274 (+ 3908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:03.546330 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:03.555827 (+ 9497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:03.565810 (+ 9983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:03.565825 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:03.565857 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:03.567582 (+ 1725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:03.568133 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:03.568137 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:03.568140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:03.568290 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:03.572327 (+ 4037us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:03.572334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:03.572516 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 03:05:03.572703 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 03:05:03.573034 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18906,"mutex_wait_us":1,"thread_start_us":18422,"threads_started":1}
I20260928 03:05:03.574630 14328 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41507
I20260928 03:05:03.574932 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.504640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:58295 (local address 127.6.192.61:41467)
0928 03:05:03.522826 (+ 18186us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:03.522837 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:03.524663 (+ 1826us) server_negotiation.cc:407] Connection header received
0928 03:05:03.526170 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:03.526176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:03.526285 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:03.526420 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:03.531255 (+ 4835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.532651 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.540395 (+ 7744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.541360 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.541533 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.556097 (+ 14564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:03.556107 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:03.556113 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:03.556168 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:03.564410 (+ 8242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:03.567721 (+ 3311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:03.567727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:03.567731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:03.567901 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:03.568751 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:03.568756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:03.568759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:03.569084 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:03.569250 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:03.574249 (+ 4999us) server_negotiation.cc:299] Negotiation successful
0928 03:05:03.574502 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17994,"thread_start_us":18229,"threads_started":1}
I20260928 03:05:03.574970 14328 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:03.575914 14328 heartbeater.cc:511] Master 127.6.192.62:41507 requested a full tablet report, sending...
I20260928 03:05:03.577656 13073 ts_manager.cc:194] Re-registered known tserver with Master: 4af241c0395a4b3aa8178f812e35329c (127.6.192.3:44019)
I20260928 03:05:03.578778 13073 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:59241
I20260928 03:05:03.579411 14329 heartbeater.cc:347] Connected to a master server at 127.6.192.61:41467
I20260928 03:05:03.579636 14329 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:03.580199 14329 heartbeater.cc:511] Master 127.6.192.61:41467 requested a full tablet report, sending...
I20260928 03:05:03.581617 13672 ts_manager.cc:194] Registered new tserver with Master: 4af241c0395a4b3aa8178f812e35329c (127.6.192.3:44019)
I20260928 03:05:03.582527 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.504659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:42959 (local address 127.6.192.60:40419)
0928 03:05:03.520066 (+ 15407us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:03.520077 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:03.523622 (+ 3545us) server_negotiation.cc:407] Connection header received
0928 03:05:03.526422 (+ 2800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:03.526430 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:03.526522 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:03.526605 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:03.532487 (+ 5882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.534254 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.548528 (+ 14274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.549569 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.549726 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.560180 (+ 10454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:03.560192 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:03.560200 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:03.560261 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:03.565494 (+ 5233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:03.574011 (+ 8517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:03.574020 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:03.574027 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:03.577342 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:03.577910 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:03.577915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:03.577919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:03.578291 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:03.578426 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:03.578975 (+ 549us) server_negotiation.cc:299] Negotiation successful
0928 03:05:03.579236 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15176,"thread_start_us":18823,"threads_started":1}
I20260928 03:05:03.583667 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.505428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40419 (local address 127.6.192.3:42959)
0928 03:05:03.521024 (+ 15596us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:03.521115 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:03.526226 (+ 5111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:03.527487 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:03.527496 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:03.531830 (+ 4334us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:03.532212 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:03.532229 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.540591 (+ 8362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.540600 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:03.548202 (+ 7602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:03.548228 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.551531 (+ 3303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.551548 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:03.551855 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.552994 (+ 1139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:03.553027 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:03.559816 (+ 6789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:03.566049 (+ 6233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:03.566058 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:03.566065 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:03.568544 (+ 2479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:03.577527 (+ 8983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:03.577532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:03.577536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:03.577765 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:03.578518 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:03.578523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:03.578673 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 03:05:03.582806 (+ 4133us) client_negotiation.cc:240] Negotiation successful
0928 03:05:03.583231 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15074,"thread_start_us":15045,"threads_started":1}
I20260928 03:05:03.585120 14327 heartbeater.cc:347] Connected to a master server at 127.6.192.60:40419
I20260928 03:05:03.585263 14327 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:03.585702 14327 heartbeater.cc:511] Master 127.6.192.60:40419 requested a full tablet report, sending...
I20260928 03:05:03.587117 13863 ts_manager.cc:194] Registered new tserver with Master: 4af241c0395a4b3aa8178f812e35329c (127.6.192.3:44019)
I20260928 03:05:03.596529 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.527686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51138 (local address 127.6.192.62:41507)
0928 03:05:03.570666 (+ 42980us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:03.570672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:03.570688 (+ 16us) server_negotiation.cc:407] Connection header received
0928 03:05:03.570737 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:03.570742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:03.570816 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:03.575754 (+ 4938us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:03.575911 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.577010 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.584643 (+ 7633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.585389 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.586882 (+ 1493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.589891 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:03.589898 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:03.589903 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:03.589955 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:03.593401 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:03.594137 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:03.594143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:03.594151 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:03.594282 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:03.594689 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:03.594695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:03.594699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:03.595106 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:03.595367 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:03.595895 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 03:05:03.596146 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42870,"thread_start_us":51123,"threads_started":1}
I20260928 03:05:03.597752 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.528324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60550 (local address 127.6.192.60:40419)
0928 03:05:03.576973 (+ 48649us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:03.576984 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:03.577003 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:05:03.577063 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:03.577068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:03.577157 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:03.577262 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:03.580621 (+ 3359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.581968 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.585977 (+ 4009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.586726 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.586879 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.590288 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:03.590294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:03.590300 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:03.590349 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:03.594581 (+ 4232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:03.595420 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:03.595425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:03.595429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:03.595626 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:03.596083 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:03.596090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:03.596094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:03.596426 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:03.596687 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:03.597155 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 03:05:03.597391 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47492,"thread_start_us":47343,"threads_started":1}
I20260928 03:05:03.599022 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.528727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47118 (local address 127.6.192.61:41467)
0928 03:05:03.575349 (+ 46622us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:03.575360 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:03.576761 (+ 1401us) server_negotiation.cc:407] Connection header received
0928 03:05:03.576974 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:03.576980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:03.577064 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:03.577193 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:03.580597 (+ 3404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.582046 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.584481 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.585245 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.585376 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.590359 (+ 4983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:03.590366 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:03.590373 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:03.590422 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:03.594865 (+ 4443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:03.595845 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:03.595851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:03.595855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:03.596038 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:03.596962 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:03.596968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:03.596971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:03.597365 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:03.597609 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:03.598360 (+ 751us) server_negotiation.cc:299] Negotiation successful
0928 03:05:03.598599 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46157,"thread_start_us":46026,"threads_started":1}
I20260928 03:05:03.601306 13073 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:05:03.608340 13073 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:05:03.671105 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.631493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:44019 (local address 127.0.0.1:45120)
0928 03:05:03.657575 (+ 26082us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:03.657609 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:03.657799 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:03.659096 (+ 1297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:03.659102 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:03.659131 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:03.659593 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:03.659613 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.662630 (+ 3017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.662641 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:03.666021 (+ 3380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:03.666049 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.669968 (+ 3919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.669975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:03.670217 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.670256 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 03:05:03.670345 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 03:05:03.670511 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25901,"thread_start_us":34734,"threads_started":1}
I20260928 03:05:03.673388 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.632094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.6.192.3:44019)
0928 03:05:03.656741 (+ 24647us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:03.656757 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:03.657977 (+ 1220us) server_negotiation.cc:407] Connection header received
0928 03:05:03.658134 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:03.658147 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:03.658507 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:03.658758 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:03.659928 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.662377 (+ 2449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.667519 (+ 5142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.669418 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.669563 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.669816 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:03.672831 (+ 3015us) server_negotiation.cc:299] Negotiation successful
0928 03:05:03.673007 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24383,"thread_start_us":24680,"threads_started":1}
I20260928 03:05:03.677227 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.643795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53120 (local address 127.6.192.2:36405)
0928 03:05:03.659135 (+ 15340us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:03.659145 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:03.667279 (+ 8134us) server_negotiation.cc:407] Connection header received
0928 03:05:03.667501 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:03.667510 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:03.667841 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:03.668081 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:03.668905 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.670503 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.674398 (+ 3895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.676024 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.676174 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.676401 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:03.676759 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 03:05:03.676946 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15151,"thread_start_us":15349,"threads_started":1}
I20260928 03:05:03.677251 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.643570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:36405 (local address 127.0.0.1:53120)
0928 03:05:03.667101 (+ 23531us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:03.667129 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:03.667296 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:03.668106 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:03.668115 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:03.668157 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:03.668666 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:03.668682 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.670691 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.670700 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:03.674201 (+ 3501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:03.674217 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.676259 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.676267 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:03.676496 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.676557 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 03:05:03.676677 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 03:05:03.676854 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23317,"thread_start_us":11687,"threads_started":1}
I20260928 03:05:03.681296 14362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.631178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:45339 (local address 127.0.0.1:49498)
0928 03:05:03.654822 (+ 23644us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:03.654858 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:03.655082 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:03.673698 (+ 18616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:03.673706 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:03.673739 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:03.674191 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:03.674204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.676376 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.676381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:03.678856 (+ 2475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:03.678870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.680391 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.680397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:03.680587 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.680629 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 03:05:03.680753 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 03:05:03.680900 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23376,"thread_start_us":35467,"threads_started":1}
I20260928 03:05:03.681326 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.631549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.6.192.1:45339)
0928 03:05:03.672107 (+ 40558us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:03.672117 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:03.672158 (+ 41us) server_negotiation.cc:407] Connection header received
0928 03:05:03.672255 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:03.672262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:03.672481 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:03.674436 (+ 1955us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:03.674678 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.675793 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.679020 (+ 3227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.680163 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.680333 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.680568 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:03.680817 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 03:05:03.680980 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40386,"thread_start_us":40247,"threads_started":1}
I20260928 03:05:03.680926 14121 tablet_service.cc:1467] Processing CreateTablet for tablet 4b2124b741a945fca9c64f80241b8c93 (DEFAULT_TABLE table=test-table [id=df64fdf4eb93435baf05b288530e0032]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:03.678041 14262 tablet_service.cc:1467] Processing CreateTablet for tablet 4b2124b741a945fca9c64f80241b8c93 (DEFAULT_TABLE table=test-table [id=df64fdf4eb93435baf05b288530e0032]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:03.682368 14121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b2124b741a945fca9c64f80241b8c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:03.683147 14262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b2124b741a945fca9c64f80241b8c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:03.684818 13976 tablet_service.cc:1467] Processing CreateTablet for tablet 4b2124b741a945fca9c64f80241b8c93 (DEFAULT_TABLE table=test-table [id=df64fdf4eb93435baf05b288530e0032]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:03.686117 13976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b2124b741a945fca9c64f80241b8c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:03.699115 14365 tablet_bootstrap.cc:492] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d: Bootstrap starting.
I20260928 03:05:03.700663 14366 tablet_bootstrap.cc:492] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512: Bootstrap starting.
I20260928 03:05:03.702596 14367 tablet_bootstrap.cc:492] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c: Bootstrap starting.
I20260928 03:05:03.705205 14365 tablet_bootstrap.cc:654] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:03.705204 14366 tablet_bootstrap.cc:654] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:03.706054 14367 tablet_bootstrap.cc:654] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:03.706745 14365 log.cc:824] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d: Log is configured to *not* fsync() on all Append() calls
I20260928 03:05:03.706745 14366 log.cc:824] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512: Log is configured to *not* fsync() on all Append() calls
I20260928 03:05:03.707157 14367 log.cc:824] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c: Log is configured to *not* fsync() on all Append() calls
I20260928 03:05:03.710525 14365 tablet_bootstrap.cc:492] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d: No bootstrap required, opened a new log
I20260928 03:05:03.710525 14366 tablet_bootstrap.cc:492] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512: No bootstrap required, opened a new log
I20260928 03:05:03.710875 14365 ts_tablet_manager.cc:1397] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260928 03:05:03.711199 14366 ts_tablet_manager.cc:1397] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260928 03:05:03.712322 14367 tablet_bootstrap.cc:492] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c: No bootstrap required, opened a new log
I20260928 03:05:03.712598 14367 ts_tablet_manager.cc:1397] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 03:05:03.721256 14367 raft_consensus.cc:348] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } } peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } }
I20260928 03:05:03.721709 14367 raft_consensus.cc:374] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:03.721833 14367 raft_consensus.cc:729] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af241c0395a4b3aa8178f812e35329c, State: Initialized, Role: FOLLOWER
I20260928 03:05:03.722553 14367 consensus_queue.cc:260] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } } peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } }
I20260928 03:05:03.723868 14328 heartbeater.cc:503] Master 127.6.192.62:41507 was elected leader, sending a full tablet report...
I20260928 03:05:03.725358 14367 ts_tablet_manager.cc:1428] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260928 03:05:03.725188 14366 raft_consensus.cc:348] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } } peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } }
I20260928 03:05:03.725822 14366 raft_consensus.cc:374] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:03.725934 14366 raft_consensus.cc:729] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b73e7c5e06f7404c928ec74316dd4512, State: Initialized, Role: FOLLOWER
I20260928 03:05:03.726786 14366 consensus_queue.cc:260] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } } peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } }
I20260928 03:05:03.731209 14365 raft_consensus.cc:348] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } } peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } }
I20260928 03:05:03.731980 14365 raft_consensus.cc:374] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:03.732122 14365 raft_consensus.cc:729] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b258d7d74c2941b983bb7bfe5d709c7d, State: Initialized, Role: FOLLOWER
I20260928 03:05:03.733031 14365 consensus_queue.cc:260] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } } peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } }
I20260928 03:05:03.735350 14365 ts_tablet_manager.cc:1428] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d: Time spent starting tablet: real 0.024s user 0.015s sys 0.001s
I20260928 03:05:03.735824 14366 ts_tablet_manager.cc:1428] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512: Time spent starting tablet: real 0.024s user 0.019s sys 0.002s
I20260928 03:05:03.737397 14187 heartbeater.cc:503] Master 127.6.192.62:41507 was elected leader, sending a full tablet report...
W20260928 03:05:03.754693 14330 tablet.cc:2369] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:05:03.827145 14189 tablet.cc:2369] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:05:03.900497 14372 raft_consensus.cc:482] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:05:03.900874 14372 raft_consensus.cc:504] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } } peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } }
I20260928 03:05:03.902693 14372 leader_election.cc:288] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b258d7d74c2941b983bb7bfe5d709c7d (127.6.192.1:45339), 4af241c0395a4b3aa8178f812e35329c (127.6.192.3:44019)
I20260928 03:05:03.910959 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.903528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:43169 (local address 127.6.192.3:44019)
0928 03:05:03.903742 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:03.903749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:03.904457 (+ 708us) server_negotiation.cc:407] Connection header received
0928 03:05:03.904674 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:03.904679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:03.904741 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:03.904852 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:03.905471 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.906329 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.908799 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.909714 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.909856 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.909968 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:03.910486 (+ 518us) server_negotiation.cc:299] Negotiation successful
0928 03:05:03.910675 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 03:05:03.911216 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.903081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:55267 (local address 127.6.192.1:45339)
0928 03:05:03.903313 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:03.903322 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:03.904219 (+ 897us) server_negotiation.cc:407] Connection header received
0928 03:05:03.904455 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:03.904459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:03.904555 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:03.904665 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:03.905242 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.905965 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.908717 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.910123 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.910280 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.910405 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:03.910776 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 03:05:03.910954 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 03:05:03.911263 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.903105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:45339 (local address 127.6.192.2:55267)
0928 03:05:03.904097 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:03.904116 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:03.904303 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:03.904681 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:03.904685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:03.904708 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:03.905097 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:03.905108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.906104 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.906110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:03.908544 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:03.908561 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.910397 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.910404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:03.910603 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.910635 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 03:05:03.910718 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 03:05:03.910850 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":45,"thread_start_us":651,"threads_started":1}
I20260928 03:05:03.911732 14375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.903359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:44019 (local address 127.6.192.2:43169)
0928 03:05:03.904339 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:03.904356 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:03.904529 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:03.904869 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:03.904874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:03.904905 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:03.905299 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:03.905308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.906466 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.906471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:03.908597 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:03.908610 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.909937 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.909943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:03.910219 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.910262 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 03:05:03.911147 (+ 885us) client_negotiation.cc:240] Negotiation successful
0928 03:05:03.911314 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":1,"thread_start_us":622,"threads_started":1}
I20260928 03:05:03.912657 13996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2124b741a945fca9c64f80241b8c93" candidate_uuid: "b73e7c5e06f7404c928ec74316dd4512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" is_pre_election: true
I20260928 03:05:03.912853 14282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2124b741a945fca9c64f80241b8c93" candidate_uuid: "b73e7c5e06f7404c928ec74316dd4512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af241c0395a4b3aa8178f812e35329c" is_pre_election: true
I20260928 03:05:03.913237 13996 raft_consensus.cc:2454] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b73e7c5e06f7404c928ec74316dd4512 in term 0.
I20260928 03:05:03.913401 14282 raft_consensus.cc:2454] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b73e7c5e06f7404c928ec74316dd4512 in term 0.
I20260928 03:05:03.914002 14077 leader_election.cc:302] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4af241c0395a4b3aa8178f812e35329c, b258d7d74c2941b983bb7bfe5d709c7d, b73e7c5e06f7404c928ec74316dd4512; no voters:
I20260928 03:05:03.914475 14372 raft_consensus.cc:2781] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:05:03.914618 14372 raft_consensus.cc:482] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:05:03.914717 14372 raft_consensus.cc:3037] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:03.917012 14372 raft_consensus.cc:504] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } } peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } }
I20260928 03:05:03.917735 14372 leader_election.cc:288] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [CANDIDATE]: Term 1 election: Requested vote from peers b258d7d74c2941b983bb7bfe5d709c7d (127.6.192.1:45339), 4af241c0395a4b3aa8178f812e35329c (127.6.192.3:44019)
I20260928 03:05:03.918567 14282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2124b741a945fca9c64f80241b8c93" candidate_uuid: "b73e7c5e06f7404c928ec74316dd4512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af241c0395a4b3aa8178f812e35329c"
I20260928 03:05:03.918685 13996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2124b741a945fca9c64f80241b8c93" candidate_uuid: "b73e7c5e06f7404c928ec74316dd4512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b258d7d74c2941b983bb7bfe5d709c7d"
I20260928 03:05:03.918802 14282 raft_consensus.cc:3037] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:03.918905 13996 raft_consensus.cc:3037] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:03.921109 14282 raft_consensus.cc:2454] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b73e7c5e06f7404c928ec74316dd4512 in term 1.
I20260928 03:05:03.921650 14075 leader_election.cc:302] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af241c0395a4b3aa8178f812e35329c, b73e7c5e06f7404c928ec74316dd4512; no voters:
I20260928 03:05:03.922062 14372 raft_consensus.cc:2781] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:05:03.923692 13996 raft_consensus.cc:2454] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b73e7c5e06f7404c928ec74316dd4512 in term 1.
I20260928 03:05:03.924190 14372 raft_consensus.cc:686] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [term 1 LEADER]: Becoming Leader. State: Replica: b73e7c5e06f7404c928ec74316dd4512, State: Running, Role: LEADER
I20260928 03:05:03.924937 14372 consensus_queue.cc:237] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } } peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } }
I20260928 03:05:03.931233 13072 catalog_manager.cc:5495] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 reported cstate change: term changed from 0 to 1, leader changed from <none> to b73e7c5e06f7404c928ec74316dd4512 (127.6.192.2). New cstate: current_term: 1 leader_uuid: "b73e7c5e06f7404c928ec74316dd4512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } health_report { overall_health: HEALTHY } } }
W20260928 03:05:03.949479 14047 tablet.cc:2369] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:05:03.992221 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:03.986595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.6.192.2:36405)
0928 03:05:03.986820 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:03.986825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:03.986844 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:05:03.986939 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:03.986943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:03.987027 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:03.987167 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:05:03.987684 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.988493 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.989942 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:03.990413 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:03.990521 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:03.991039 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:05:03.991619 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:05:03.991733 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:03.991921 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 03:05:03.991993 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 03:05:04.001376 13996 raft_consensus.cc:1261] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d [term 1 FOLLOWER]: Refusing update from remote peer b73e7c5e06f7404c928ec74316dd4512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:05:04.001621 14282 raft_consensus.cc:1261] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c [term 1 FOLLOWER]: Refusing update from remote peer b73e7c5e06f7404c928ec74316dd4512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:05:04.002617 14376 consensus_queue.cc:1035] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:05:04.003067 14372 consensus_queue.cc:1035] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:05:04.038728 13072 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:05:04.039036 13072 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:05:04.046630 13072 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a2716d3b7d545dd82f8fadedefd92d9: Sending DeleteTablet for 3 replicas of tablet 4b2124b741a945fca9c64f80241b8c93
I20260928 03:05:04.047932 13976 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b2124b741a945fca9c64f80241b8c93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:04 UTC) from {username='oryx', principal='oryx/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:49498
I20260928 03:05:04.047989 14121 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b2124b741a945fca9c64f80241b8c93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:04 UTC) from {username='oryx', principal='oryx/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:53120
I20260928 03:05:04.047983 14262 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b2124b741a945fca9c64f80241b8c93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:04 UTC) from {username='oryx', principal='oryx/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:45120
I20260928 03:05:04.061432 14390 tablet_replica.cc:330] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c: stopping tablet replica
I20260928 03:05:04.062059 14390 raft_consensus.cc:2229] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:05:04.062685 14390 raft_consensus.cc:2258] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:04.063681 14391 tablet_replica.cc:330] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d: stopping tablet replica
I20260928 03:05:04.064146 14391 raft_consensus.cc:2229] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:05:04.064757 14391 raft_consensus.cc:2258] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:04.066236 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 13041
I20260928 03:05:04.066370 14389 tablet_replica.cc:330] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512: stopping tablet replica
I20260928 03:05:04.066928 14389 raft_consensus.cc:2229] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:05:04.067480 13103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:04.068207 14391 ts_tablet_manager.cc:1905] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:04.073704 14390 ts_tablet_manager.cc:1905] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:04.076035 14391 ts_tablet_manager.cc:1918] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:05:04.076228 14391 log.cc:1196] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/4b2124b741a945fca9c64f80241b8c93
I20260928 03:05:04.076951 14391 ts_tablet_manager.cc:1939] T 4b2124b741a945fca9c64f80241b8c93 P b258d7d74c2941b983bb7bfe5d709c7d: Deleting consensus metadata
I20260928 03:05:04.080132 14389 raft_consensus.cc:2258] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:04.083626 14389 ts_tablet_manager.cc:1905] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:04.091181 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:04.056177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47128 (local address 127.6.192.61:41467)
0928 03:05:04.056450 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:04.056459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:04.064211 (+ 7752us) server_negotiation.cc:407] Connection header received
0928 03:05:04.064580 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:04.064590 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:04.064689 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:04.064845 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:04.067913 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:04.068942 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:04.074049 (+ 5107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:04.074737 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:04.074911 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:04.079859 (+ 4948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:04.079866 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:04.079882 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:04.079933 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:04.084170 (+ 4237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:04.085204 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:04.085212 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:04.085216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:04.085380 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:04.085857 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:04.085863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:04.085866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:04.086234 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:04.086543 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:04.090494 (+ 3951us) server_negotiation.cc:299] Negotiation successful
0928 03:05:04.090801 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 03:05:04.091784 14389 ts_tablet_manager.cc:1918] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:05:04.091984 14389 log.cc:1196] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/4b2124b741a945fca9c64f80241b8c93
I20260928 03:05:04.092620 14389 ts_tablet_manager.cc:1939] T 4b2124b741a945fca9c64f80241b8c93 P b73e7c5e06f7404c928ec74316dd4512: Deleting consensus metadata
I20260928 03:05:04.101893 14390 ts_tablet_manager.cc:1918] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:05:04.102128 14390 log.cc:1196] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/4b2124b741a945fca9c64f80241b8c93
I20260928 03:05:04.102720 14390 ts_tablet_manager.cc:1939] T 4b2124b741a945fca9c64f80241b8c93 P 4af241c0395a4b3aa8178f812e35329c: Deleting consensus metadata
I20260928 03:05:04.109311 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:04.064545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.6.192.60:40419)
0928 03:05:04.065180 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:04.065187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:04.088000 (+ 22813us) server_negotiation.cc:407] Connection header received
0928 03:05:04.088262 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:04.088269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:04.088357 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:04.088431 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:04.092879 (+ 4448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:04.093969 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:04.096116 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:04.096689 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:04.096826 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:04.101087 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:04.101096 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:04.101102 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:04.101162 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:04.105303 (+ 4141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:04.106361 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:04.106367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:04.106380 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:04.106557 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:04.107554 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:04.107560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:04.107564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:04.107961 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:04.108251 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:04.108621 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 03:05:04.108878 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446}
I20260928 03:05:04.218469 13058 catalog_manager.cc:4841] TS 4af241c0395a4b3aa8178f812e35329c (127.6.192.3:44019): tablet 4b2124b741a945fca9c64f80241b8c93 (table test-table [id=df64fdf4eb93435baf05b288530e0032]) successfully deleted
I20260928 03:05:04.219583 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 13041
I20260928 03:05:04.267788 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:04.251172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47142 (local address 127.6.192.61:41467)
0928 03:05:04.251366 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:04.251371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:04.251662 (+ 291us) server_negotiation.cc:407] Connection header received
0928 03:05:04.251856 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:04.251862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:04.251973 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:04.252083 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:04.254062 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:04.255193 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:04.257350 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:04.257830 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:04.257964 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:04.261648 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:04.261654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:04.261659 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:04.261700 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:04.264693 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:04.265512 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:04.265518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:04.265521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:04.265669 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:04.266093 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:04.266101 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:04.266106 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:04.266479 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:04.266740 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:04.267161 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 03:05:04.267402 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 03:05:04.269881 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:04.246413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60568 (local address 127.6.192.60:40419)
0928 03:05:04.249267 (+ 2854us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:04.249273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:04.250963 (+ 1690us) server_negotiation.cc:407] Connection header received
0928 03:05:04.251281 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:04.251287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:04.251411 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:04.251526 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:04.254133 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:04.255193 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:04.258250 (+ 3057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:04.258944 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:04.259084 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:04.262890 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:04.262898 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:04.262903 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:04.262947 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:04.266204 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:04.266959 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:04.266966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:04.266973 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:04.267125 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:04.268314 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:04.268319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:04.268322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:04.268690 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:04.269252 (+ 562us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:04.269373 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 03:05:04.269539 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2692}
I20260928 03:05:05.033563 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:05.032499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.1:53803)
0928 03:05:05.033074 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:05.033352 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":309,"threads_started":1}
W20260928 03:05:05.034240 14042 heartbeater.cc:644] Failed to heartbeat to 127.6.192.62:41507 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
I20260928 03:05:05.034499 14401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:05.033643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.2:41105)
0928 03:05:05.034173 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:05.034314 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":325,"threads_started":1}
W20260928 03:05:05.034874 14187 heartbeater.cc:644] Failed to heartbeat to 127.6.192.62:41507 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
I20260928 03:05:05.038483 14402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:05.037543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.3:46795)
0928 03:05:05.038095 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:05.038248 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":324,"threads_started":1}
W20260928 03:05:05.038939 14328 heartbeater.cc:644] Failed to heartbeat to 127.6.192.62:41507 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
I20260928 03:05:05.546582 14403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a2716d3b7d545dd82f8fadedefd92d9)
I20260928 03:05:05.546938 14403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } }
I20260928 03:05:05.547585 14404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a2716d3b7d545dd82f8fadedefd92d9)
I20260928 03:05:05.547873 14404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } }
I20260928 03:05:05.548844 14403 leader_election.cc:288] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a2716d3b7d545dd82f8fadedefd92d9 (127.6.192.62:41507), 04b9a59f6fd94c6892c5278980919e61 (127.6.192.60:40419)
I20260928 03:05:05.549427 14404 leader_election.cc:288] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a2716d3b7d545dd82f8fadedefd92d9 (127.6.192.62:41507), 569125e0394b44a8a3d58d5fb58d6821 (127.6.192.61:41467)
I20260928 03:05:05.550892 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:05.548999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:51572)
0928 03:05:05.550094 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:05.550344 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":920,"thread_start_us":1157,"threads_started":1}
W20260928 03:05:05.551215 13658 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
I20260928 03:05:05.552155 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:05.549523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:51576)
0928 03:05:05.551639 (+ 2116us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:05.551865 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1972,"thread_start_us":2127,"threads_started":1}
W20260928 03:05:05.553097 13849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
I20260928 03:05:05.554275 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:05.553681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:51592)
0928 03:05:05.553929 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:05.554038 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260928 03:05:05.554677 13658 leader_election.cc:334] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a2716d3b7d545dd82f8fadedefd92d9 (127.6.192.62:41507): Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
I20260928 03:05:05.556581 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:05.556034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:51598)
0928 03:05:05.556236 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:05.556405 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 03:05:05.556821 13849 leader_election.cc:334] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a2716d3b7d545dd82f8fadedefd92d9 (127.6.192.62:41507): Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
I20260928 03:05:05.560649 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:05.549548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60572 (local address 127.6.192.60:40419)
0928 03:05:05.553280 (+ 3732us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:05.553286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:05.553305 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:05:05.553373 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:05.553380 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:05.553454 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:05.554361 (+ 907us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:05.554655 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:05.555915 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:05.558648 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:05.559532 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:05.559672 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:05.559831 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:05.560111 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 03:05:05.560241 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3586,"thread_start_us":3447,"threads_started":1}
I20260928 03:05:05.560730 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:05.552044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:41467 (local address 127.0.0.1:47146)
0928 03:05:05.552356 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:05.552381 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:05.552504 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:05.553713 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:05.553720 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:05.553785 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:05.554039 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:05.554053 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:05.556134 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:05.556142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:05.558469 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:05.558484 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:05.559849 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:05.559856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:05.560066 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:05.560091 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 03:05:05.560208 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 03:05:05.560371 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":681,"threads_started":1}
I20260928 03:05:05.561312 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:05.552556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.6.192.61:41467)
0928 03:05:05.553303 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:05.553331 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:05.553361 (+ 30us) server_negotiation.cc:407] Connection header received
0928 03:05:05.553433 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:05.553438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:05.553539 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:05.554194 (+ 655us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:05.554470 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:05.555939 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:05.558684 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:05.559607 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:05.559828 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:05.559950 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:05.560243 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 03:05:05.560387 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":609,"thread_start_us":777,"threads_started":1}
I20260928 03:05:05.562320 14405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:05.549064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40419 (local address 127.0.0.1:60572)
0928 03:05:05.550526 (+ 1462us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:05.550554 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:05.551049 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:05.553626 (+ 2577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:05.553635 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:05.553676 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:05.554151 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:05.554167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:05.556076 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:05.556086 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:05.558466 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:05.558483 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:05.559723 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:05.559729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:05.559948 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:05.559972 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 03:05:05.560085 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 03:05:05.560223 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1291,"mutex_wait_us":27,"thread_start_us":685,"threads_started":1}
I20260928 03:05:05.562965 13682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04b9a59f6fd94c6892c5278980919e61" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "569125e0394b44a8a3d58d5fb58d6821" is_pre_election: true
I20260928 03:05:05.563114 13873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "569125e0394b44a8a3d58d5fb58d6821" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "04b9a59f6fd94c6892c5278980919e61" is_pre_election: true
I20260928 03:05:05.563279 13682 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04b9a59f6fd94c6892c5278980919e61 in term 1.
I20260928 03:05:05.563496 13873 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 569125e0394b44a8a3d58d5fb58d6821 in term 1.
I20260928 03:05:05.563912 13849 leader_election.cc:302] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04b9a59f6fd94c6892c5278980919e61, 569125e0394b44a8a3d58d5fb58d6821; no voters: 7a2716d3b7d545dd82f8fadedefd92d9
I20260928 03:05:05.564043 13656 leader_election.cc:302] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04b9a59f6fd94c6892c5278980919e61, 569125e0394b44a8a3d58d5fb58d6821; no voters: 7a2716d3b7d545dd82f8fadedefd92d9
I20260928 03:05:05.564286 14404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 03:05:05.564426 14404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a2716d3b7d545dd82f8fadedefd92d9)
I20260928 03:05:05.564538 14404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 1 FOLLOWER]: Advancing to term 2
I20260928 03:05:05.564724 14403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 03:05:05.564864 14403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a2716d3b7d545dd82f8fadedefd92d9)
I20260928 03:05:05.564977 14403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 1 FOLLOWER]: Advancing to term 2
I20260928 03:05:05.568125 14404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } }
I20260928 03:05:05.568307 14403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } }
I20260928 03:05:05.569036 14404 leader_election.cc:288] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [CANDIDATE]: Term 2 election: Requested vote from peers 7a2716d3b7d545dd82f8fadedefd92d9 (127.6.192.62:41507), 569125e0394b44a8a3d58d5fb58d6821 (127.6.192.61:41467)
I20260928 03:05:05.569325 14403 leader_election.cc:288] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [CANDIDATE]: Term 2 election: Requested vote from peers 7a2716d3b7d545dd82f8fadedefd92d9 (127.6.192.62:41507), 04b9a59f6fd94c6892c5278980919e61 (127.6.192.60:40419)
I20260928 03:05:05.569579 13873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "569125e0394b44a8a3d58d5fb58d6821" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "04b9a59f6fd94c6892c5278980919e61"
I20260928 03:05:05.569900 13873 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 569125e0394b44a8a3d58d5fb58d6821 in current term 2: Already voted for candidate 04b9a59f6fd94c6892c5278980919e61 in this term.
I20260928 03:05:05.569872 13682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04b9a59f6fd94c6892c5278980919e61" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "569125e0394b44a8a3d58d5fb58d6821"
I20260928 03:05:05.570044 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:05.569270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:51608)
0928 03:05:05.569663 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:05.569812 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264}
I20260928 03:05:05.570165 13682 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 04b9a59f6fd94c6892c5278980919e61 in current term 2: Already voted for candidate 569125e0394b44a8a3d58d5fb58d6821 in this term.
I20260928 03:05:05.570194 14405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:05.569383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:51610)
0928 03:05:05.569910 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:05.570006 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408}
I20260928 03:05:05.571384 14405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:05.570986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:51646)
0928 03:05:05.571175 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:05.571263 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260928 03:05:05.571622 13658 leader_election.cc:334] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7a2716d3b7d545dd82f8fadedefd92d9 (127.6.192.62:41507): Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
I20260928 03:05:05.571728 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:05.570792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:51636)
0928 03:05:05.570945 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:05.571567 (+ 622us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 03:05:05.571779 13658 leader_election.cc:302] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 569125e0394b44a8a3d58d5fb58d6821; no voters: 04b9a59f6fd94c6892c5278980919e61, 7a2716d3b7d545dd82f8fadedefd92d9
W20260928 03:05:05.571906 13849 leader_election.cc:334] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7a2716d3b7d545dd82f8fadedefd92d9 (127.6.192.62:41507): Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
I20260928 03:05:05.572070 13849 leader_election.cc:302] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 04b9a59f6fd94c6892c5278980919e61; no voters: 569125e0394b44a8a3d58d5fb58d6821, 7a2716d3b7d545dd82f8fadedefd92d9
I20260928 03:05:05.572120 14403 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 03:05:05.572391 14404 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 03:05:06.036087 14413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:06.035087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.1:53075)
0928 03:05:06.035720 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:06.035838 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":397,"threads_started":1}
I20260928 03:05:06.036811 14414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:06.035576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.2:36457)
0928 03:05:06.036566 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:06.036661 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":879,"thread_start_us":777,"threads_started":1}
I20260928 03:05:06.040441 14415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:06.039668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.3:48337)
0928 03:05:06.040169 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:06.040270 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":317,"threads_started":1}
I20260928 03:05:07.038322 14416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.037197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.1:60249)
0928 03:05:07.037965 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.038088 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":508,"threads_started":1}
I20260928 03:05:07.039240 14417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.037742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.2:49115)
0928 03:05:07.038915 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.039004 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1048,"thread_start_us":927,"threads_started":1}
I20260928 03:05:07.042277 14418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.041384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.3:60697)
0928 03:05:07.041919 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.042025 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":245,"threads_started":1}
I20260928 03:05:07.142247 14419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:05:07.142578 14419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } }
I20260928 03:05:07.143352 14419 leader_election.cc:288] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a2716d3b7d545dd82f8fadedefd92d9 (127.6.192.62:41507), 569125e0394b44a8a3d58d5fb58d6821 (127.6.192.61:41467)
I20260928 03:05:07.144562 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.143554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:51860)
0928 03:05:07.144147 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.144280 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":320,"threads_started":1}
I20260928 03:05:07.144748 13682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04b9a59f6fd94c6892c5278980919e61" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "569125e0394b44a8a3d58d5fb58d6821" is_pre_election: true
I20260928 03:05:07.145099 13682 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04b9a59f6fd94c6892c5278980919e61 in term 2.
I20260928 03:05:07.145593 13849 leader_election.cc:302] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04b9a59f6fd94c6892c5278980919e61, 569125e0394b44a8a3d58d5fb58d6821; no voters:
I20260928 03:05:07.145962 14419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 03:05:07.146096 14419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:05:07.146217 14419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 2 FOLLOWER]: Advancing to term 3
I20260928 03:05:07.146546 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.146114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:51872)
0928 03:05:07.146297 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.146401 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 03:05:07.146790 13849 leader_election.cc:334] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7a2716d3b7d545dd82f8fadedefd92d9 (127.6.192.62:41507): Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
I20260928 03:05:07.148845 14419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } }
I20260928 03:05:07.149467 14419 leader_election.cc:288] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [CANDIDATE]: Term 3 election: Requested vote from peers 7a2716d3b7d545dd82f8fadedefd92d9 (127.6.192.62:41507), 569125e0394b44a8a3d58d5fb58d6821 (127.6.192.61:41467)
I20260928 03:05:07.150000 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.149604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:51876)
0928 03:05:07.149745 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.149839 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 03:05:07.150275 13682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04b9a59f6fd94c6892c5278980919e61" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "569125e0394b44a8a3d58d5fb58d6821"
I20260928 03:05:07.150532 13682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 2 FOLLOWER]: Advancing to term 3
I20260928 03:05:07.151122 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.150774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:51890)
0928 03:05:07.150938 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.151015 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 03:05:07.151372 13849 leader_election.cc:334] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7a2716d3b7d545dd82f8fadedefd92d9 (127.6.192.62:41507): Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
I20260928 03:05:07.153239 13682 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04b9a59f6fd94c6892c5278980919e61 in term 3.
I20260928 03:05:07.153750 13849 leader_election.cc:302] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04b9a59f6fd94c6892c5278980919e61, 569125e0394b44a8a3d58d5fb58d6821; no voters: 7a2716d3b7d545dd82f8fadedefd92d9
I20260928 03:05:07.154174 14419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 03:05:07.154786 14419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [term 3 LEADER]: Becoming Leader. State: Replica: 04b9a59f6fd94c6892c5278980919e61, State: Running, Role: LEADER
I20260928 03:05:07.155200 14419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } }
I20260928 03:05:07.157260 14422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04b9a59f6fd94c6892c5278980919e61. Latest consensus state: current_term: 3 leader_uuid: "04b9a59f6fd94c6892c5278980919e61" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } } }
I20260928 03:05:07.157562 14422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [sys.catalog]: This master's current role is: LEADER
I20260928 03:05:07.158206 14424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:05:07.162308 14424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:05:07.163311 14424 catalog_manager.cc:1259] Loaded cluster ID: 3459c6c9524d4c0eab848e651e9c2e9c
I20260928 03:05:07.163439 14424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:05:07.164830 14424 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:05:07.165804 14424 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: Loaded TSK: 0
I20260928 03:05:07.167017 14424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:05:07.248590 13862 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:05:07.256094 13862 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:05:07.262169 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.261707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:51928)
0928 03:05:07.261930 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.262017 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 03:05:07.262226 13682 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [term 3 FOLLOWER]: Refusing update from remote peer 04b9a59f6fd94c6892c5278980919e61: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 03:05:07.263366 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.262905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:51938)
0928 03:05:07.263082 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.263195 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 03:05:07.263371 14419 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 03:05:07.263726 13849 consensus_peers.cc:487] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 -> Peer 7a2716d3b7d545dd82f8fadedefd92d9 (127.6.192.62:41507): Couldn't send request to peer 7a2716d3b7d545dd82f8fadedefd92d9. Status: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 03:05:07.267158 14426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04b9a59f6fd94c6892c5278980919e61. Latest consensus state: current_term: 3 leader_uuid: "04b9a59f6fd94c6892c5278980919e61" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } } }
I20260928 03:05:07.267606 14426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:05:07.269435 14426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "04b9a59f6fd94c6892c5278980919e61" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } } }
I20260928 03:05:07.269699 14426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569125e0394b44a8a3d58d5fb58d6821 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:05:07.273424 14427 mvcc.cc:205] Tried to move back new op lower bound from 7334153040937103360 to 7334153040511553536. Current Snapshot: MvccSnapshot[applied={T|T < 7334153040937103360 or (T in {7334153040937103360})}]
I20260928 03:05:07.274317 14430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "04b9a59f6fd94c6892c5278980919e61" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7a2716d3b7d545dd82f8fadedefd92d9" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41507 } } peers { permanent_uuid: "569125e0394b44a8a3d58d5fb58d6821" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 41467 } attrs { promote: false } } peers { permanent_uuid: "04b9a59f6fd94c6892c5278980919e61" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40419 } attrs { promote: false } } }
I20260928 03:05:07.274739 14430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61 [sys.catalog]: This master's current role is: LEADER
I20260928 03:05:07.307614 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.282942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.6.192.3:44019)
0928 03:05:07.299990 (+ 17048us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:07.300001 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:07.300032 (+ 31us) server_negotiation.cc:407] Connection header received
0928 03:05:07.300129 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:07.300138 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:07.300256 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:07.301135 (+ 879us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:07.301421 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.302929 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.305268 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.306333 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.306546 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:07.306661 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:07.307077 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 03:05:07.307237 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16836,"thread_start_us":16909,"threads_started":1}
I20260928 03:05:07.307644 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.282718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:44019 (local address 127.0.0.1:45128)
0928 03:05:07.283055 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.283083 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:07.283241 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:07.300505 (+ 17264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:07.300514 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:07.300560 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:07.300941 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:07.300958 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.303081 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.303085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:07.305064 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:07.305080 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.306651 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.306659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:07.306865 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:07.306917 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 03:05:07.307023 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 03:05:07.307205 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220}
I20260928 03:05:07.308956 14262 tablet_service.cc:1467] Processing CreateTablet for tablet c4fc580d834a4c8899162748cac7c34d (DEFAULT_TABLE table=test-table [id=9a162dd5f2f74d3a84faef4ccc1447a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:07.309489 14262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4fc580d834a4c8899162748cac7c34d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:07.310721 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.284094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53136 (local address 127.6.192.2:36405)
0928 03:05:07.303830 (+ 19736us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:07.303837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:07.303857 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:05:07.303925 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:07.303932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:07.304009 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:07.304126 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:07.304771 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.305866 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.308470 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.309659 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.309807 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:07.309911 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:07.310224 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 03:05:07.310380 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19577,"thread_start_us":19432,"threads_started":1}
I20260928 03:05:07.310830 14435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.283655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:36405 (local address 127.0.0.1:53136)
0928 03:05:07.303096 (+ 19441us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.303124 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:07.303262 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:07.304174 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:07.304178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:07.304200 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:07.304564 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:07.304574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.306043 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.306050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:07.308247 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:07.308260 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.309851 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.309858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:07.310045 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:07.310086 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 03:05:07.310261 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 03:05:07.310457 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16564,"thread_start_us":16695,"threads_started":1}
I20260928 03:05:07.311597 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.283151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:45339 (local address 127.0.0.1:49508)
0928 03:05:07.299518 (+ 16367us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.299548 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:07.299738 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:07.302115 (+ 2377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:07.302122 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:07.302162 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:07.302925 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:07.302940 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.306422 (+ 3482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.306428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:07.308978 (+ 2550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:07.308992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.310809 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.310820 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:07.311028 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:07.311065 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 03:05:07.311147 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 03:05:07.311267 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16146,"thread_start_us":15948,"threads_started":1}
I20260928 03:05:07.311789 14121 tablet_service.cc:1467] Processing CreateTablet for tablet c4fc580d834a4c8899162748cac7c34d (DEFAULT_TABLE table=test-table [id=9a162dd5f2f74d3a84faef4ccc1447a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:07.312104 14434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.283616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49508 (local address 127.6.192.1:45339)
0928 03:05:07.292415 (+ 8799us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:07.292424 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:07.299656 (+ 7232us) server_negotiation.cc:407] Connection header received
0928 03:05:07.299889 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:07.299894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:07.300000 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:07.300109 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:07.305297 (+ 5188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.306274 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.309255 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.310410 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.310540 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:07.310658 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:07.311567 (+ 909us) server_negotiation.cc:299] Negotiation successful
0928 03:05:07.311721 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8628,"thread_start_us":8501,"threads_started":1}
I20260928 03:05:07.312263 14121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4fc580d834a4c8899162748cac7c34d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:07.315761 13976 tablet_service.cc:1467] Processing CreateTablet for tablet c4fc580d834a4c8899162748cac7c34d (DEFAULT_TABLE table=test-table [id=9a162dd5f2f74d3a84faef4ccc1447a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:07.316232 14437 tablet_bootstrap.cc:492] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c: Bootstrap starting.
I20260928 03:05:07.316257 13976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4fc580d834a4c8899162748cac7c34d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:07.317574 14437 tablet_bootstrap.cc:654] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:07.319937 14438 tablet_bootstrap.cc:492] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512: Bootstrap starting.
I20260928 03:05:07.320418 14437 tablet_bootstrap.cc:492] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c: No bootstrap required, opened a new log
I20260928 03:05:07.320586 14437 ts_tablet_manager.cc:1397] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 03:05:07.321442 14437 raft_consensus.cc:348] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } } peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } }
I20260928 03:05:07.321759 14437 raft_consensus.cc:374] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:07.321815 14438 tablet_bootstrap.cc:654] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:07.321866 14437 raft_consensus.cc:729] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af241c0395a4b3aa8178f812e35329c, State: Initialized, Role: FOLLOWER
I20260928 03:05:07.322150 14437 consensus_queue.cc:260] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } } peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } }
I20260928 03:05:07.323191 14437 ts_tablet_manager.cc:1428] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 03:05:07.324653 14418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.323989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.3:56669)
0928 03:05:07.324314 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.324450 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260928 03:05:07.326184 14418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.325595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.3:37875)
0928 03:05:07.325847 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.325943 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260928 03:05:07.327232 14327 heartbeater.cc:503] Master 127.6.192.60:40419 was elected leader, sending a full tablet report...
I20260928 03:05:07.332113 14442 tablet_bootstrap.cc:492] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d: Bootstrap starting.
I20260928 03:05:07.333501 14438 tablet_bootstrap.cc:492] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512: No bootstrap required, opened a new log
I20260928 03:05:07.333679 14438 ts_tablet_manager.cc:1397] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.009s
I20260928 03:05:07.333797 14442 tablet_bootstrap.cc:654] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:07.334421 14438 raft_consensus.cc:348] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } } peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } }
I20260928 03:05:07.334666 14438 raft_consensus.cc:374] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:07.334822 14438 raft_consensus.cc:729] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b73e7c5e06f7404c928ec74316dd4512, State: Initialized, Role: FOLLOWER
I20260928 03:05:07.335152 14438 consensus_queue.cc:260] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } } peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } }
I20260928 03:05:07.336159 14438 ts_tablet_manager.cc:1428] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 03:05:07.336994 14417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.336531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.2:36139)
0928 03:05:07.336682 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.336782 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 03:05:07.337368 14442 tablet_bootstrap.cc:492] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d: No bootstrap required, opened a new log
I20260928 03:05:07.337589 14442 ts_tablet_manager.cc:1397] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 03:05:07.338303 14417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.337605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.2:48995)
0928 03:05:07.338012 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.338093 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291}
I20260928 03:05:07.338464 14186 heartbeater.cc:503] Master 127.6.192.60:40419 was elected leader, sending a full tablet report...
I20260928 03:05:07.338512 14442 raft_consensus.cc:348] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } } peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } }
I20260928 03:05:07.338898 14442 raft_consensus.cc:374] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:07.339062 14442 raft_consensus.cc:729] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b258d7d74c2941b983bb7bfe5d709c7d, State: Initialized, Role: FOLLOWER
I20260928 03:05:07.339403 14442 consensus_queue.cc:260] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } } peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } }
I20260928 03:05:07.340579 14442 ts_tablet_manager.cc:1428] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 03:05:07.341606 14416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.341128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.1:36847)
0928 03:05:07.341307 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.341425 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 03:05:07.342656 14041 heartbeater.cc:503] Master 127.6.192.60:40419 was elected leader, sending a full tablet report...
I20260928 03:05:07.342857 14416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.342311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.1:46769)
0928 03:05:07.342584 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.342685 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260928 03:05:07.403823 14444 raft_consensus.cc:482] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:05:07.404206 14444 raft_consensus.cc:504] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } } peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } }
I20260928 03:05:07.405022 14444 leader_election.cc:288] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af241c0395a4b3aa8178f812e35329c (127.6.192.3:44019), b258d7d74c2941b983bb7bfe5d709c7d (127.6.192.1:45339)
I20260928 03:05:07.405714 13996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4fc580d834a4c8899162748cac7c34d" candidate_uuid: "b73e7c5e06f7404c928ec74316dd4512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" is_pre_election: true
I20260928 03:05:07.405766 14282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4fc580d834a4c8899162748cac7c34d" candidate_uuid: "b73e7c5e06f7404c928ec74316dd4512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af241c0395a4b3aa8178f812e35329c" is_pre_election: true
I20260928 03:05:07.406028 13996 raft_consensus.cc:2454] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b73e7c5e06f7404c928ec74316dd4512 in term 0.
I20260928 03:05:07.406028 14282 raft_consensus.cc:2454] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b73e7c5e06f7404c928ec74316dd4512 in term 0.
I20260928 03:05:07.406587 14075 leader_election.cc:302] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4af241c0395a4b3aa8178f812e35329c, b258d7d74c2941b983bb7bfe5d709c7d, b73e7c5e06f7404c928ec74316dd4512; no voters:
I20260928 03:05:07.406947 14444 raft_consensus.cc:2781] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:05:07.407068 14444 raft_consensus.cc:482] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:05:07.407148 14444 raft_consensus.cc:3037] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:07.409343 14444 raft_consensus.cc:504] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } } peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } }
I20260928 03:05:07.409992 14444 leader_election.cc:288] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [CANDIDATE]: Term 1 election: Requested vote from peers 4af241c0395a4b3aa8178f812e35329c (127.6.192.3:44019), b258d7d74c2941b983bb7bfe5d709c7d (127.6.192.1:45339)
I20260928 03:05:07.410625 14282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4fc580d834a4c8899162748cac7c34d" candidate_uuid: "b73e7c5e06f7404c928ec74316dd4512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af241c0395a4b3aa8178f812e35329c"
I20260928 03:05:07.410640 13996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4fc580d834a4c8899162748cac7c34d" candidate_uuid: "b73e7c5e06f7404c928ec74316dd4512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b258d7d74c2941b983bb7bfe5d709c7d"
I20260928 03:05:07.410858 14282 raft_consensus.cc:3037] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:07.410866 13996 raft_consensus.cc:3037] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:07.412995 13996 raft_consensus.cc:2454] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b73e7c5e06f7404c928ec74316dd4512 in term 1.
I20260928 03:05:07.413158 14282 raft_consensus.cc:2454] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b73e7c5e06f7404c928ec74316dd4512 in term 1.
I20260928 03:05:07.413571 14077 leader_election.cc:302] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b258d7d74c2941b983bb7bfe5d709c7d, b73e7c5e06f7404c928ec74316dd4512; no voters:
I20260928 03:05:07.413906 14444 raft_consensus.cc:2781] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:05:07.414481 14444 raft_consensus.cc:686] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [term 1 LEADER]: Becoming Leader. State: Replica: b73e7c5e06f7404c928ec74316dd4512, State: Running, Role: LEADER
I20260928 03:05:07.414829 14444 consensus_queue.cc:237] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } } peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } }
I20260928 03:05:07.415925 14417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.415386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.2:39261)
0928 03:05:07.415592 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.415714 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 03:05:07.417727 13862 catalog_manager.cc:5495] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 reported cstate change: term changed from 0 to 1, leader changed from <none> to b73e7c5e06f7404c928ec74316dd4512 (127.6.192.2). New cstate: current_term: 1 leader_uuid: "b73e7c5e06f7404c928ec74316dd4512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b73e7c5e06f7404c928ec74316dd4512" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36405 } health_report { overall_health: HEALTHY } } }
I20260928 03:05:07.457558 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.452279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53150 (local address 127.6.192.2:36405)
0928 03:05:07.452637 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:07.452645 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:07.452667 (+ 22us) server_negotiation.cc:407] Connection header received
0928 03:05:07.452760 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:07.452765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:07.452851 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:07.452937 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:05:07.453304 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.454153 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.455583 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.456099 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.456216 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:07.456676 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:05:07.456822 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:05:07.456910 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:07.457070 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 03:05:07.457144 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 03:05:07.461081 13996 raft_consensus.cc:1261] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d [term 1 FOLLOWER]: Refusing update from remote peer b73e7c5e06f7404c928ec74316dd4512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:05:07.461171 14282 raft_consensus.cc:1261] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c [term 1 FOLLOWER]: Refusing update from remote peer b73e7c5e06f7404c928ec74316dd4512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:05:07.461855 14446 consensus_queue.cc:1035] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b258d7d74c2941b983bb7bfe5d709c7d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:05:07.462200 14444 consensus_queue.cc:1035] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af241c0395a4b3aa8178f812e35329c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:05:07.466161 14416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.465039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.1:34081)
0928 03:05:07.465762 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.465915 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550}
I20260928 03:05:07.466863 14418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.465443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.3:59745)
0928 03:05:07.466526 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.466646 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":926}
I20260928 03:05:07.475261 14416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.474606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.1:39867)
0928 03:05:07.474966 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.475074 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222}
I20260928 03:05:07.475950 14417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.475036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.2:53109)
0928 03:05:07.475560 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.475694 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309}
I20260928 03:05:07.476410 14451 mvcc.cc:205] Tried to move back new op lower bound from 7334153041754107904 to 7334153041573969920. Current Snapshot: MvccSnapshot[applied={T|T < 7334153041754107904}]
I20260928 03:05:07.483688 13862 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:05:07.483924 13862 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:05:07.489593 14417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.489082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.2:56147)
0928 03:05:07.489299 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.489409 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 03:05:07.491840 13862 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04b9a59f6fd94c6892c5278980919e61: Sending DeleteTablet for 3 replicas of tablet c4fc580d834a4c8899162748cac7c34d
I20260928 03:05:07.493016 14262 tablet_service.cc:1514] Processing DeleteTablet for tablet c4fc580d834a4c8899162748cac7c34d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:07 UTC) from {username='oryx', principal='oryx/127.6.192.60@KRBTEST.COM'} at 127.0.0.1:45128
I20260928 03:05:07.493057 13976 tablet_service.cc:1514] Processing DeleteTablet for tablet c4fc580d834a4c8899162748cac7c34d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:07 UTC) from {username='oryx', principal='oryx/127.6.192.60@KRBTEST.COM'} at 127.0.0.1:49508
I20260928 03:05:07.493830 14458 tablet_replica.cc:330] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c: stopping tablet replica
I20260928 03:05:07.494104 14458 raft_consensus.cc:2229] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:05:07.494275 14459 tablet_replica.cc:330] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d: stopping tablet replica
I20260928 03:05:07.494508 14458 raft_consensus.cc:2258] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:07.494587 14459 raft_consensus.cc:2229] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:05:07.494984 14459 raft_consensus.cc:2258] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:07.495242 14418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.494428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.6.192.3:55647)
0928 03:05:07.494897 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.495014 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346}
I20260928 03:05:07.495963 14458 ts_tablet_manager.cc:1905] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:07.496630 14459 ts_tablet_manager.cc:1905] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:07.497633 14121 tablet_service.cc:1514] Processing DeleteTablet for tablet c4fc580d834a4c8899162748cac7c34d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:07 UTC) from {username='oryx', principal='oryx/127.6.192.60@KRBTEST.COM'} at 127.0.0.1:53136
I20260928 03:05:07.498080 14120 tablet_service.cc:1514] Processing DeleteTablet for tablet c4fc580d834a4c8899162748cac7c34d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:07 UTC) from {username='oryx', principal='oryx/127.6.192.60@KRBTEST.COM'} at 127.0.0.1:53136
I20260928 03:05:07.500932 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 13841
I20260928 03:05:07.503515 14458 ts_tablet_manager.cc:1918] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:05:07.503814 14458 log.cc:1196] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/c4fc580d834a4c8899162748cac7c34d
I20260928 03:05:07.504568 14458 ts_tablet_manager.cc:1939] T c4fc580d834a4c8899162748cac7c34d P 4af241c0395a4b3aa8178f812e35329c: Deleting consensus metadata
I20260928 03:05:07.506121 13848 catalog_manager.cc:4841] TS 4af241c0395a4b3aa8178f812e35329c (127.6.192.3:44019): tablet c4fc580d834a4c8899162748cac7c34d (table test-table [id=9a162dd5f2f74d3a84faef4ccc1447a2]) successfully deleted
I20260928 03:05:07.513374 14460 tablet_replica.cc:330] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512: stopping tablet replica
I20260928 03:05:07.513803 14460 raft_consensus.cc:2229] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:05:07.514389 14460 raft_consensus.cc:2258] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:07.514550 14459 ts_tablet_manager.cc:1918] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:05:07.514853 14459 log.cc:1196] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/c4fc580d834a4c8899162748cac7c34d
I20260928 03:05:07.515848 14460 ts_tablet_manager.cc:1905] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:07.516110 14459 ts_tablet_manager.cc:1939] T c4fc580d834a4c8899162748cac7c34d P b258d7d74c2941b983bb7bfe5d709c7d: Deleting consensus metadata
I20260928 03:05:07.517478 13850 catalog_manager.cc:4841] TS b258d7d74c2941b983bb7bfe5d709c7d (127.6.192.1:45339): tablet c4fc580d834a4c8899162748cac7c34d (table test-table [id=9a162dd5f2f74d3a84faef4ccc1447a2]) successfully deleted
I20260928 03:05:07.522135 14460 ts_tablet_manager.cc:1918] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:05:07.522400 14460 log.cc:1196] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/c4fc580d834a4c8899162748cac7c34d
I20260928 03:05:07.522907 14460 ts_tablet_manager.cc:1939] T c4fc580d834a4c8899162748cac7c34d P b73e7c5e06f7404c928ec74316dd4512: Deleting consensus metadata
I20260928 03:05:07.524201 13849 catalog_manager.cc:4841] TS b73e7c5e06f7404c928ec74316dd4512 (127.6.192.2:36405): tablet c4fc580d834a4c8899162748cac7c34d (table test-table [id=9a162dd5f2f74d3a84faef4ccc1447a2]) successfully deleted
W20260928 03:05:07.524778 13849 catalog_manager.cc:4808] TS b73e7c5e06f7404c928ec74316dd4512 (127.6.192.2:36405): delete failed for tablet c4fc580d834a4c8899162748cac7c34d because the tablet was not found. No further retry: Not found: Tablet not found: c4fc580d834a4c8899162748cac7c34d
I20260928 03:05:07.534094 14434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.502040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49520 (local address 127.6.192.1:45339)
0928 03:05:07.502231 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:07.502238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:07.515211 (+ 12973us) server_negotiation.cc:407] Connection header received
0928 03:05:07.515560 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:07.515567 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:07.515663 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:07.515742 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:07.518062 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.519131 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.521435 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.522163 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.522349 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:07.526604 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:07.526629 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:07.526653 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:07.526746 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:07.530741 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:07.531726 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:07.531735 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:07.531737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:07.531895 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:07.532352 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:07.532356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:07.532358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:07.532751 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:07.533110 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:07.533452 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 03:05:07.533707 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 03:05:07.535379 14036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:07.631722 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 13841
I20260928 03:05:07.652282 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 14053
I20260928 03:05:07.667889 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.653369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53152 (local address 127.6.192.2:36405)
0928 03:05:07.653604 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:07.653612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:07.653633 (+ 21us) server_negotiation.cc:407] Connection header received
0928 03:05:07.653865 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:07.653877 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:07.653958 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:07.654077 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:07.655586 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.656428 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.657889 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.658350 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.658498 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:07.661439 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:07.661456 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:07.661472 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:07.661511 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:07.664658 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:07.665539 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:07.665555 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:07.665558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:07.665694 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:07.666103 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:07.666108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:07.666111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:07.666520 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:07.666833 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:07.667159 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 03:05:07.667430 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 03:05:07.668581 14181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:07.682058 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.681480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:51946)
0928 03:05:07.681719 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.681848 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 03:05:07.683126 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.682722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41507 (local address 127.0.0.1:51954)
0928 03:05:07.682876 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:07.682965 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 03:05:07.794011 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 14053
I20260928 03:05:07.816843 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 14199
I20260928 03:05:07.833091 14464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:07.817667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45136 (local address 127.6.192.3:44019)
0928 03:05:07.818271 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:07.818275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:07.818294 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:05:07.818360 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:07.818364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:07.818439 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:07.818537 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:07.820536 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.821321 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.822773 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:07.823216 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:07.823329 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:07.826345 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:07.826363 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:07.826380 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:07.826469 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:07.829618 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:07.830504 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:07.830517 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:07.830520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:07.830748 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:07.831361 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:07.831366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:07.831368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:07.831869 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:07.832137 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:07.832610 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 03:05:07.832817 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":348,"threads_started":1}
I20260928 03:05:07.833768 14322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:07.929237 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 14199
I20260928 03:05:07.948781 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 13535
I20260928 03:05:07.949923 13702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:08.073278 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 13535
I20260928 03:05:08.094791 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 13634
I20260928 03:05:08.095876 13893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:08.221987 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 13634
2026-09-28T03:05:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11765 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:05:08 dist-test-slave-fjww krb5kdc[14470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:05:08 dist-test-slave-fjww krb5kdc[14470](info): set up 2 sockets
Sep 28 03:05:08 dist-test-slave-fjww krb5kdc[14470](info): commencing operation
krb5kdc: starting...
W20260928 03:05:09.404983 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:05:09 dist-test-slave-fjww krb5kdc[14470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:05:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:05:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.62.keytab.
Entry for principal oryx/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.62.keytab.
I20260928 03:05:09.648327 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:34851
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.62.keytab
--principal=oryx/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:34851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:05:10.003822 14483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:10.004151 14483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:10.004334 14483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:10.011999 14483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:05:10.012125 14483 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 03:05:10.012192 14483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:10.012291 14483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:10.012395 14483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:05:10.012485 14483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:05:10.023015 14483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:34851
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.62.keytab
--principal=oryx/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:34851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:10.024490 14483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:10.025959 14483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:10.039628 14488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:10.039667 14491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:10.039713 14489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:10.040692 14483 server_base.cc:1034] running on GCE node
I20260928 03:05:10.041567 14483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:10.043210 14483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:10.044510 14483 hybrid_clock.cc:648] HybridClock initialized: now 1790564710044434 us; error 68 us; skew 500 ppm
I20260928 03:05:10.044996 14483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:10 dist-test-slave-fjww krb5kdc[14470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564710, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:10.048692 14483 init.cc:377] Logged in from keytab as oryx/127.6.192.62@KRBTEST.COM (short username oryx)
I20260928 03:05:10.050398 14483 webserver.cc:460] Webserver started at http://127.6.192.62:40661/ using document root <none> and password file <none>
I20260928 03:05:10.051162 14483 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:10.051263 14483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:10.051648 14483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:10.054965 14483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "924cd42c531346eaa908271ad04ca8a8"
format_stamp: "Formatted at 2026-09-28 03:05:10 on dist-test-slave-fjww"
server_key: "9213a1d31127f20bb62c445657825aa5"
server_key_iv: "28a11d20c5b34d94541dcec5449793bc"
server_key_version: "encryptionkey@0"
I20260928 03:05:10.055977 14483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "924cd42c531346eaa908271ad04ca8a8"
format_stamp: "Formatted at 2026-09-28 03:05:10 on dist-test-slave-fjww"
server_key: "9213a1d31127f20bb62c445657825aa5"
server_key_iv: "28a11d20c5b34d94541dcec5449793bc"
server_key_version: "encryptionkey@0"
I20260928 03:05:10.062126 14483 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 03:05:10.066095 14498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:10.067848 14483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:05:10.068024 14483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "924cd42c531346eaa908271ad04ca8a8"
format_stamp: "Formatted at 2026-09-28 03:05:10 on dist-test-slave-fjww"
server_key: "9213a1d31127f20bb62c445657825aa5"
server_key_iv: "28a11d20c5b34d94541dcec5449793bc"
server_key_version: "encryptionkey@0"
I20260928 03:05:10.068241 14483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:10.083082 14483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:10.088150 14483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:10.088416 14483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:10.114426 14483 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:34851
I20260928 03:05:10.114440 14559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:34851 every 8 connection(s)
I20260928 03:05:10.116150 14483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:05:10.120589 14560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:10.122645 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 14483
I20260928 03:05:10.122874 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:05:10.123605 6912 external_mini_cluster.cc:1428] Setting key b8398bf93b0dd8219c066e7c7da8708f
I20260928 03:05:10.135076 14560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8: Bootstrap starting.
I20260928 03:05:10.139834 14560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8: Neither blocks nor log segments found. Creating new log.
Sep 28 03:05:10 dist-test-slave-fjww krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM
I20260928 03:05:10.141314 14560 log.cc:824] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8: Log is configured to *not* fsync() on all Append() calls
I20260928 03:05:10.145265 14560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8: No bootstrap required, opened a new log
I20260928 03:05:10.151605 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:10.126054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35160 (local address 127.6.192.62:34851)
0928 03:05:10.126997 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:10.127010 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:10.127056 (+ 46us) server_negotiation.cc:407] Connection header received
0928 03:05:10.129005 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:10.129031 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:10.129479 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:10.129841 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:10.132091 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:10.133598 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:10.135735 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:10.136650 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:10.136834 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:10.142579 (+ 5745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:10.142596 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:10.142625 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:10.142683 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:10.146469 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:10.147583 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:10.147591 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:10.147599 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:10.147800 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:10.148253 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:10.148257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:10.148259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:10.148904 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:10.149130 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:10.149699 (+ 569us) server_negotiation.cc:299] Negotiation successful
0928 03:05:10.150179 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":463,"threads_started":1}
I20260928 03:05:10.155131 14560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924cd42c531346eaa908271ad04ca8a8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34851 } }
I20260928 03:05:10.155607 14560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:10.155721 14560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924cd42c531346eaa908271ad04ca8a8, State: Initialized, Role: FOLLOWER
I20260928 03:05:10.156350 14560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924cd42c531346eaa908271ad04ca8a8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34851 } }
I20260928 03:05:10.156618 14560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:05:10.156761 14560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:05:10.156951 14560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:10.159206 14560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924cd42c531346eaa908271ad04ca8a8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34851 } }
I20260928 03:05:10.159830 14560 leader_election.cc:302] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 924cd42c531346eaa908271ad04ca8a8; no voters:
I20260928 03:05:10.160507 14560 leader_election.cc:288] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:05:10.160706 14565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:05:10.161782 14565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8 [term 1 LEADER]: Becoming Leader. State: Replica: 924cd42c531346eaa908271ad04ca8a8, State: Running, Role: LEADER
I20260928 03:05:10.162472 14565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924cd42c531346eaa908271ad04ca8a8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34851 } }
I20260928 03:05:10.162853 14560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:05:10.166227 14566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "924cd42c531346eaa908271ad04ca8a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924cd42c531346eaa908271ad04ca8a8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34851 } } }
I20260928 03:05:10.166761 14566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8 [sys.catalog]: This master's current role is: LEADER
I20260928 03:05:10.166222 14567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 924cd42c531346eaa908271ad04ca8a8. Latest consensus state: current_term: 1 leader_uuid: "924cd42c531346eaa908271ad04ca8a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924cd42c531346eaa908271ad04ca8a8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34851 } } }
I20260928 03:05:10.168051 14567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8 [sys.catalog]: This master's current role is: LEADER
W20260928 03:05:10.176462 14581 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 03:05:10.176627 14581 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 03:05:10.176748 14573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:05:10.179040 14573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:05:10.189101 14573 catalog_manager.cc:1347] Generated new cluster ID: 070bfbe8248441768958a9bd49a7ea90
I20260928 03:05:10.189231 14573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:05:10.210714 14573 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:05:10.212417 14573 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:05:10.218698 14573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8: Generated new TSK 0
I20260928 03:05:10.219893 14573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.1.keytab.
Entry for principal oryx/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.1.keytab.
I20260928 03:05:10.280408 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34851
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.1.keytab
--principal=oryx/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:05:10.632490 14586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:10.632853 14586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:10.633037 14586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:10.640858 14586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:10.641057 14586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:10.641206 14586 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 03:05:10.641345 14586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:05:10.651885 14586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.1.keytab
--principal=oryx/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.192.62:34851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:10.653307 14586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:10.654770 14586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:10.669696 14592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:10.670063 14591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:10.669705 14594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:10.670461 14586 server_base.cc:1034] running on GCE node
I20260928 03:05:10.671275 14586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:10.672430 14586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:10.673712 14586 hybrid_clock.cc:648] HybridClock initialized: now 1790564710673632 us; error 58 us; skew 500 ppm
I20260928 03:05:10.674094 14586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:10 dist-test-slave-fjww krb5kdc[14470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564710, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:10.677650 14586 init.cc:377] Logged in from keytab as oryx/127.6.192.1@KRBTEST.COM (short username oryx)
I20260928 03:05:10.679160 14586 webserver.cc:460] Webserver started at http://127.6.192.1:38945/ using document root <none> and password file <none>
I20260928 03:05:10.679975 14586 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:10.680068 14586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:10.680364 14586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:10.683408 14586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "a8fe39b38ba14c088a1d2297d2f8c999"
format_stamp: "Formatted at 2026-09-28 03:05:10 on dist-test-slave-fjww"
server_key: "675261ed7d84a4e16dbe4aa4778fb6f7"
server_key_iv: "62edfa095fb89990791fbb3d8dde18e3"
server_key_version: "encryptionkey@0"
I20260928 03:05:10.684269 14586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "a8fe39b38ba14c088a1d2297d2f8c999"
format_stamp: "Formatted at 2026-09-28 03:05:10 on dist-test-slave-fjww"
server_key: "675261ed7d84a4e16dbe4aa4778fb6f7"
server_key_iv: "62edfa095fb89990791fbb3d8dde18e3"
server_key_version: "encryptionkey@0"
I20260928 03:05:10.689965 14586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:05:10.694238 14601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:10.695932 14586 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:05:10.696128 14586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "a8fe39b38ba14c088a1d2297d2f8c999"
format_stamp: "Formatted at 2026-09-28 03:05:10 on dist-test-slave-fjww"
server_key: "675261ed7d84a4e16dbe4aa4778fb6f7"
server_key_iv: "62edfa095fb89990791fbb3d8dde18e3"
server_key_version: "encryptionkey@0"
I20260928 03:05:10.696489 14586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:10.721459 14586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:10.726626 14586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:10.726980 14586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:10.730073 14586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:10.730226 14586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:10.730441 14586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:10.730531 14586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 03:05:10 dist-test-slave-fjww krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564710, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM
I20260928 03:05:10.759838 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:10.731057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:46267 (local address 127.6.192.62:34851)
0928 03:05:10.731703 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:10.731711 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:10.733111 (+ 1400us) server_negotiation.cc:407] Connection header received
0928 03:05:10.735293 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:10.735299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:10.735411 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:10.735571 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:10.738799 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:10.739817 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:10.743533 (+ 3716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:10.744107 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:10.744267 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:10.750375 (+ 6108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:10.750388 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:10.750396 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:10.750455 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:10.754996 (+ 4541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:10.756358 (+ 1362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:10.756369 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:10.756375 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:10.756570 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:10.757377 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:10.757385 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:10.757388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:10.757878 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:10.758043 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:10.758944 (+ 901us) server_negotiation.cc:299] Negotiation successful
0928 03:05:10.759291 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":417,"threads_started":1}
I20260928 03:05:10.761107 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:10.731892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34851 (local address 127.6.192.1:46267)
0928 03:05:10.732869 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:10.732940 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:10.734949 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:10.735958 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:10.735973 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:10.737148 (+ 1175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:10.738422 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:10.738505 (+ 83us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:10.740052 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:10.740059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:10.743128 (+ 3069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:10.743161 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:10.744438 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:10.744446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:10.744886 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:10.745852 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:10.745904 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:10.749950 (+ 4046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:10.755256 (+ 5306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:10.755270 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:10.755298 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:10.756187 (+ 889us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:10.756747 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:10.756754 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:10.756762 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:10.757151 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:10.758059 (+ 908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:10.758072 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:10.758496 (+ 424us) client_negotiation.cc:769] Sending connection context
0928 03:05:10.758738 (+ 242us) client_negotiation.cc:240] Negotiation successful
0928 03:05:10.759166 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":441,"threads_started":1}
I20260928 03:05:10.777889 14586 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:46095
I20260928 03:05:10.777953 14721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:46095 every 8 connection(s)
I20260928 03:05:10.779711 14586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:05:10.782820 14722 heartbeater.cc:347] Connected to a master server at 127.6.192.62:34851
I20260928 03:05:10.783102 14722 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:10.783915 14722 heartbeater.cc:511] Master 127.6.192.62:34851 requested a full tablet report, sending...
I20260928 03:05:10.786477 14515 ts_manager.cc:194] Registered new tserver with Master: a8fe39b38ba14c088a1d2297d2f8c999 (127.6.192.1:46095)
I20260928 03:05:10.788264 14515 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:46267
I20260928 03:05:10.788463 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:10.769942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:36659 (local address 127.6.192.62:34851)
0928 03:05:10.770557 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:10.770567 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:10.771489 (+ 922us) server_negotiation.cc:407] Connection header received
0928 03:05:10.771786 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:10.771795 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:10.771896 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:10.772019 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:10.773827 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:10.775176 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:10.777659 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:10.778406 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:10.778530 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:10.781662 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:10.781669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:10.781672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:10.781709 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:10.785275 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:10.786110 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:10.786117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:10.786121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:10.786302 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:10.786836 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:10.786842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:10.786845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:10.787159 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:10.787334 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:10.787827 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 03:05:10.788069 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420}
I20260928 03:05:10.788519 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:10.770066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34851 (local address 127.6.192.1:36659)
0928 03:05:10.771277 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:10.771294 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:10.771598 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:10.772082 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:10.772089 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:10.773009 (+ 920us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:10.773596 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:10.773619 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:10.775415 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:10.775421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:10.777376 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:10.777412 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:10.778632 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:10.778640 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:10.778842 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:10.779578 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:10.779607 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:10.781359 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:10.785519 (+ 4160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:10.785523 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:10.785526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:10.785951 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:10.786482 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:10.786486 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:10.786488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:10.786700 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:10.787285 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:10.787290 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:10.787561 (+ 271us) client_negotiation.cc:769] Sending connection context
0928 03:05:10.787700 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 03:05:10.788052 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"thread_start_us":893,"threads_started":1}
I20260928 03:05:10.789716 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 14586
I20260928 03:05:10.789953 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:05:10.790697 6912 external_mini_cluster.cc:1428] Setting key 4d784bc757ae8ecb4794608e5da59cdd
WARNING: no policy specified for oryx/127.6.192.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.192.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.2.keytab.
Entry for principal oryx/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.2.keytab.
I20260928 03:05:10.847529 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34851
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.2.keytab
--principal=oryx/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:05:11.185305 14726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:11.185616 14726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:11.185778 14726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:11.193576 14726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:11.193688 14726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:11.193826 14726 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 03:05:11.193948 14726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:05:11.204532 14726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.2.keytab
--principal=oryx/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.192.62:34851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:11.205890 14726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:11.207278 14726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:11.222126 14732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:11.222195 14734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:11.222098 14731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:11.223152 14726 server_base.cc:1034] running on GCE node
I20260928 03:05:11.223974 14726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:11.225028 14726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:11.226393 14726 hybrid_clock.cc:648] HybridClock initialized: now 1790564711226289 us; error 78 us; skew 500 ppm
I20260928 03:05:11.226797 14726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:11 dist-test-slave-fjww krb5kdc[14470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564711, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:11.230582 14726 init.cc:377] Logged in from keytab as oryx/127.6.192.2@KRBTEST.COM (short username oryx)
I20260928 03:05:11.232438 14726 webserver.cc:460] Webserver started at http://127.6.192.2:38729/ using document root <none> and password file <none>
I20260928 03:05:11.233212 14726 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:11.233372 14726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:11.233721 14726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:11.237074 14726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "7e7518965ca94f1182344204d56b18a0"
format_stamp: "Formatted at 2026-09-28 03:05:11 on dist-test-slave-fjww"
server_key: "143645bc9aa41a52f01f6a3a1022f5a9"
server_key_iv: "6c8d252a81e298d7527f858dd1fe7b7b"
server_key_version: "encryptionkey@0"
I20260928 03:05:11.237979 14726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "7e7518965ca94f1182344204d56b18a0"
format_stamp: "Formatted at 2026-09-28 03:05:11 on dist-test-slave-fjww"
server_key: "143645bc9aa41a52f01f6a3a1022f5a9"
server_key_iv: "6c8d252a81e298d7527f858dd1fe7b7b"
server_key_version: "encryptionkey@0"
I20260928 03:05:11.243903 14726 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 03:05:11.247813 14741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:11.249233 14726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:05:11.249425 14726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "7e7518965ca94f1182344204d56b18a0"
format_stamp: "Formatted at 2026-09-28 03:05:11 on dist-test-slave-fjww"
server_key: "143645bc9aa41a52f01f6a3a1022f5a9"
server_key_iv: "6c8d252a81e298d7527f858dd1fe7b7b"
server_key_version: "encryptionkey@0"
I20260928 03:05:11.249665 14726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:11.279371 14726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:11.284492 14726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:11.284795 14726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:11.287631 14726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:11.287716 14726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:11.287911 14726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:11.287987 14726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 03:05:11 dist-test-slave-fjww krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564711, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM
I20260928 03:05:11.318706 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:11.288777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:45737 (local address 127.6.192.62:34851)
0928 03:05:11.288985 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:11.288992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:11.290781 (+ 1789us) server_negotiation.cc:407] Connection header received
0928 03:05:11.294198 (+ 3417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:11.294208 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:11.294444 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:11.294602 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:11.297700 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.298710 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.302095 (+ 3385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.302568 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.302687 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:11.308637 (+ 5950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:11.308647 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:11.308655 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:11.308708 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:11.312316 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:11.313601 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:11.313611 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:11.313616 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:11.313804 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:11.314533 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:11.314537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:11.314539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:11.314949 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:11.315120 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:11.317953 (+ 2833us) server_negotiation.cc:299] Negotiation successful
0928 03:05:11.318258 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 03:05:11.319095 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:11.289474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34851 (local address 127.6.192.2:45737)
0928 03:05:11.290526 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:11.290594 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:11.293672 (+ 3078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:11.295352 (+ 1680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:11.295369 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:11.296358 (+ 989us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:11.297470 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:11.297504 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.298926 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.298933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:11.301849 (+ 2916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:11.301881 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.302867 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.302877 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:11.303297 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:11.304370 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:11.304419 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:11.308234 (+ 3815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:11.312495 (+ 4261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:11.312505 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:11.312528 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:11.313330 (+ 802us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:11.313959 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:11.313968 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:11.313972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:11.314343 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:11.315206 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:11.315215 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:11.315883 (+ 668us) client_negotiation.cc:769] Sending connection context
0928 03:05:11.316340 (+ 457us) client_negotiation.cc:240] Negotiation successful
0928 03:05:11.317032 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":395,"threads_started":1}
I20260928 03:05:11.338479 14726 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:46481
I20260928 03:05:11.338512 14860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:46481 every 8 connection(s)
I20260928 03:05:11.340143 14726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:05:11.342540 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 14726
I20260928 03:05:11.342728 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:05:11.343338 6912 external_mini_cluster.cc:1428] Setting key 3e1c6f96b08e3078da3540103a08df83
I20260928 03:05:11.344018 14861 heartbeater.cc:347] Connected to a master server at 127.6.192.62:34851
I20260928 03:05:11.344296 14861 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:11.345141 14861 heartbeater.cc:511] Master 127.6.192.62:34851 requested a full tablet report, sending...
I20260928 03:05:11.346717 14514 ts_manager.cc:194] Registered new tserver with Master: 7e7518965ca94f1182344204d56b18a0 (127.6.192.2:46481)
I20260928 03:05:11.348089 14514 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:45737
I20260928 03:05:11.349242 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:11.331679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:51733 (local address 127.6.192.62:34851)
0928 03:05:11.331932 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:11.331941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:11.332775 (+ 834us) server_negotiation.cc:407] Connection header received
0928 03:05:11.333040 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:11.333047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:11.333152 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:11.333287 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:11.335249 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.336486 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.338910 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.339334 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.339519 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:11.342215 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:11.342224 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:11.342230 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:11.342276 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:11.345990 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:11.346815 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:11.346821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:11.346824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:11.346992 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:11.347529 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:11.347534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:11.347537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:11.347997 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:11.348127 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:11.348763 (+ 636us) server_negotiation.cc:299] Negotiation successful
0928 03:05:11.349017 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 03:05:11.350077 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:11.331023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34851 (local address 127.6.192.2:51733)
0928 03:05:11.332607 (+ 1584us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:11.332632 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:11.332838 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:11.333350 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:11.333358 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:11.334323 (+ 965us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:11.334941 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:11.334964 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.336643 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.336651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:11.338726 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:11.338744 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.339579 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.339587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:11.339774 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:11.340564 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:11.340588 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:11.341931 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:11.346153 (+ 4222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:11.346161 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:11.346170 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:11.346670 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:11.347202 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:11.347206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:11.347210 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:11.347363 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:11.348232 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:11.348237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:11.348428 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 03:05:11.349420 (+ 992us) client_negotiation.cc:240] Negotiation successful
0928 03:05:11.349750 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1339,"thread_start_us":1118,"threads_started":1}
WARNING: no policy specified for oryx/127.6.192.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.192.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.3.keytab.
Entry for principal oryx/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.3.keytab.
I20260928 03:05:11.396163 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34851
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.3.keytab
--principal=oryx/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:05:11.734634 14865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:11.734943 14865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:11.735121 14865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:11.742956 14865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:11.743086 14865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:11.743230 14865 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 03:05:11.743530 14865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:05:11.753993 14865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.3.keytab
--principal=oryx/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.192.62:34851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:11.755529 14865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:11.756932 14865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:11.772095 14871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:11.772065 14873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:11.772037 14870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:11.773010 14865 server_base.cc:1034] running on GCE node
I20260928 03:05:11.773862 14865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:11.774976 14865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:11.776257 14865 hybrid_clock.cc:648] HybridClock initialized: now 1790564711776171 us; error 62 us; skew 500 ppm
I20260928 03:05:11.776660 14865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:11 dist-test-slave-fjww krb5kdc[14470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564711, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:11.781080 14865 init.cc:377] Logged in from keytab as oryx/127.6.192.3@KRBTEST.COM (short username oryx)
I20260928 03:05:11.783155 14865 webserver.cc:460] Webserver started at http://127.6.192.3:46645/ using document root <none> and password file <none>
I20260928 03:05:11.784325 14865 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:11.784480 14865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:11.784849 14865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:11.788102 14865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "7e08753d65844d5d85150d55bcc41271"
format_stamp: "Formatted at 2026-09-28 03:05:11 on dist-test-slave-fjww"
server_key: "150c93216d25c6c59a198d8d3d43493f"
server_key_iv: "66578d5a3b04ab4a3fc1da05e3655d60"
server_key_version: "encryptionkey@0"
I20260928 03:05:11.789135 14865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "7e08753d65844d5d85150d55bcc41271"
format_stamp: "Formatted at 2026-09-28 03:05:11 on dist-test-slave-fjww"
server_key: "150c93216d25c6c59a198d8d3d43493f"
server_key_iv: "66578d5a3b04ab4a3fc1da05e3655d60"
server_key_version: "encryptionkey@0"
I20260928 03:05:11.794485 14722 heartbeater.cc:503] Master 127.6.192.62:34851 was elected leader, sending a full tablet report...
I20260928 03:05:11.795173 14865 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 03:05:11.799351 14880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:11.801275 14865 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:05:11.801440 14865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "7e08753d65844d5d85150d55bcc41271"
format_stamp: "Formatted at 2026-09-28 03:05:11 on dist-test-slave-fjww"
server_key: "150c93216d25c6c59a198d8d3d43493f"
server_key_iv: "66578d5a3b04ab4a3fc1da05e3655d60"
server_key_version: "encryptionkey@0"
I20260928 03:05:11.801707 14865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:11.823869 14865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:11.828925 14865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:11.829241 14865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:11.832454 14865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:11.832581 14865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:11.832777 14865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:11.832864 14865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 03:05:11 dist-test-slave-fjww krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564711, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM
I20260928 03:05:11.863565 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:11.834315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34851 (local address 127.6.192.3:40053)
0928 03:05:11.835318 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:11.835370 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:11.837623 (+ 2253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:11.838645 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:11.838660 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:11.839868 (+ 1208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:11.841213 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:11.841309 (+ 96us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.843200 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.843205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:11.845839 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:11.845870 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.847296 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.847307 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:11.847669 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:11.848685 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:11.848737 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:11.852923 (+ 4186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:11.857546 (+ 4623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:11.857563 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:11.857597 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:11.858491 (+ 894us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:11.859029 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:11.859037 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:11.859042 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:11.859290 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:11.860065 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:11.860074 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:11.860574 (+ 500us) client_negotiation.cc:769] Sending connection context
0928 03:05:11.860965 (+ 391us) client_negotiation.cc:240] Negotiation successful
0928 03:05:11.861637 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":443,"threads_started":1}
I20260928 03:05:11.865528 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:11.833604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:40053 (local address 127.6.192.62:34851)
0928 03:05:11.833827 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:11.833832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:11.835594 (+ 1762us) server_negotiation.cc:407] Connection header received
0928 03:05:11.837984 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:11.837990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:11.838095 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:11.838219 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:11.841576 (+ 3357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.842941 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.846075 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.846930 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.847113 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:11.853466 (+ 6353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:11.853481 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:11.853484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:11.853530 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:11.857340 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:11.858702 (+ 1362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:11.858708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:11.858710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:11.858879 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:11.859481 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:11.859513 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:11.859519 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:11.859872 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:11.863625 (+ 3753us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:11.864556 (+ 931us) server_negotiation.cc:299] Negotiation successful
0928 03:05:11.864995 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 03:05:11.885598 14865 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:33831
I20260928 03:05:11.885617 14999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:33831 every 8 connection(s)
I20260928 03:05:11.886945 14865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:05:11.889699 15000 heartbeater.cc:347] Connected to a master server at 127.6.192.62:34851
I20260928 03:05:11.890097 15000 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:11.890976 15000 heartbeater.cc:511] Master 127.6.192.62:34851 requested a full tablet report, sending...
I20260928 03:05:11.891135 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 14865
I20260928 03:05:11.891425 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:05:11.892074 6912 external_mini_cluster.cc:1428] Setting key 3f26b90b470fecefb033a7a717696315
I20260928 03:05:11.892942 14978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:11.874404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34851 (local address 127.6.192.3:37501)
0928 03:05:11.875914 (+ 1510us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:11.875939 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:11.876166 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:11.876605 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:11.876611 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:11.877408 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:11.878035 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:11.878058 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.879524 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.879533 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:11.881815 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:11.881840 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.883037 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.883047 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:11.883236 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:11.884103 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:11.884123 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:11.885268 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:11.889386 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:11.889392 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:11.889396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:11.890140 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:11.890727 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:11.890732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:11.890737 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:11.890939 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:11.891691 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:11.891697 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:11.891934 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 03:05:11.892102 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 03:05:11.892500 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"thread_start_us":560,"threads_started":1}
I20260928 03:05:11.892971 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:11.874284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:37501 (local address 127.6.192.62:34851)
0928 03:05:11.874534 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:11.874540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:11.876079 (+ 1539us) server_negotiation.cc:407] Connection header received
0928 03:05:11.876313 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:11.876319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:11.876415 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:11.876550 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:11.878279 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.879290 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.882061 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.882771 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.882955 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:11.885529 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:11.885536 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:11.885540 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:11.885577 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:11.889232 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:11.890405 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:11.890416 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:11.890422 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:11.890589 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:11.891112 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:11.891117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:11.891120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:11.891552 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:11.891705 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:11.892391 (+ 686us) server_negotiation.cc:299] Negotiation successful
0928 03:05:11.892611 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 03:05:11.894323 14515 ts_manager.cc:194] Registered new tserver with Master: 7e08753d65844d5d85150d55bcc41271 (127.6.192.3:33831)
I20260928 03:05:11.895525 14515 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:40053
I20260928 03:05:11.895602 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 03:05:11 dist-test-slave-fjww krb5kdc[14470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 03:05:11 dist-test-slave-fjww krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:11 dist-test-slave-fjww krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 03:05:11.952328 14610 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:11.939939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40848 (local address 127.6.192.62:34851)
0928 03:05:11.940132 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:11.940137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:11.940159 (+ 22us) server_negotiation.cc:407] Connection header received
0928 03:05:11.940395 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:11.940401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:11.940472 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:11.940575 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:11.942330 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.943253 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.944769 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.945323 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.945449 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:11.951948 (+ 6499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40848: BlockingRecv error: recv got EOF from 127.0.0.1:40848 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 03:05:11 dist-test-slave-fjww krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM
I20260928 03:05:11.972571 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:11.956348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.6.192.62:34851)
0928 03:05:11.956584 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:11.956593 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:11.956846 (+ 253us) server_negotiation.cc:407] Connection header received
0928 03:05:11.957078 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:11.957084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:11.957177 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:11.957286 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:11.958613 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.959372 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.960756 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:11.961193 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:11.961340 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:11.966094 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:11.966101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:11.966104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:11.966149 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:11.969445 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:11.970341 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:11.970346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:11.970349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:11.970506 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:11.971002 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:11.971006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:11.971008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:11.971311 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:11.971634 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:11.971986 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 03:05:11.972273 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 03:05:11.977506 14515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:05:11.984648 14515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:05:12.013015 15020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:11.998159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:33831 (local address 127.0.0.1:36768)
0928 03:05:11.999281 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:11.999304 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:11.999467 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:12.002402 (+ 2935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:12.002410 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:12.002443 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:12.002868 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:12.002893 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.006086 (+ 3193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.006096 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:12.009123 (+ 3027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:12.009146 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.011671 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.011680 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:12.011971 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:12.012062 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 03:05:12.012189 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 03:05:12.012434 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"mutex_wait_us":4,"thread_start_us":894,"threads_started":1}
I20260928 03:05:12.014446 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:11.998800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.6.192.3:33831)
0928 03:05:12.000785 (+ 1985us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:12.000798 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:12.000841 (+ 43us) server_negotiation.cc:407] Connection header received
0928 03:05:12.000967 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:12.001844 (+ 877us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:12.002151 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:12.003313 (+ 1162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:12.003798 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.005904 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.009469 (+ 3565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.011402 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.013566 (+ 2164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:12.013854 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:12.013929 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 03:05:12.014119 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1793,"thread_start_us":1607,"threads_started":1}
I20260928 03:05:12.018605 14930 tablet_service.cc:1467] Processing CreateTablet for tablet 673ee9b4e9f843a098b2779777ee3b07 (DEFAULT_TABLE table=test-table [id=9559ce9652214b2ab6fa68661ec46da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:12.019831 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:11.997620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:46095 (local address 127.0.0.1:44444)
0928 03:05:11.998428 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:11.998472 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:11.998678 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:12.005969 (+ 7291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:12.005977 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:12.006016 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:12.006660 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:12.006680 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.011212 (+ 4532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.011222 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:12.014209 (+ 2987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:12.014234 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.018753 (+ 4519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.018764 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:12.019047 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:12.019133 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 03:05:12.019217 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 03:05:12.019386 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":24,"thread_start_us":344,"threads_started":1}
I20260928 03:05:12.020347 14930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 673ee9b4e9f843a098b2779777ee3b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:12.020872 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:11.997779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44444 (local address 127.6.192.1:46095)
0928 03:05:12.005088 (+ 7309us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:12.005102 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:12.005140 (+ 38us) server_negotiation.cc:407] Connection header received
0928 03:05:12.005259 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:12.005272 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:12.005582 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:12.006989 (+ 1407us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:12.008865 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.010978 (+ 2113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.016296 (+ 5318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.018435 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.019187 (+ 752us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:12.019506 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:12.019615 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 03:05:12.020370 (+ 755us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7072,"thread_start_us":16998,"threads_started":1}
I20260928 03:05:12.021989 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:11.999825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.6.192.2:46481)
0928 03:05:12.002019 (+ 2194us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:12.002034 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:12.002071 (+ 37us) server_negotiation.cc:407] Connection header received
0928 03:05:12.002204 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:12.002215 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:12.002554 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:12.002810 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:12.012912 (+ 10102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.014369 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.016835 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.018524 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.018653 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:12.018859 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:12.021529 (+ 2670us) server_negotiation.cc:299] Negotiation successful
0928 03:05:12.021715 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1972,"thread_start_us":1762,"threads_started":1}
I20260928 03:05:12.022765 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:11.999177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:46481 (local address 127.0.0.1:48898)
0928 03:05:12.000000 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:12.000021 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:12.000137 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:12.012166 (+ 12029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:12.012172 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:12.012208 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:12.012634 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:12.012653 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.014580 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.014586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:12.016631 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:12.016649 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.019062 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.019068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:12.021276 (+ 2208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:12.021312 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 03:05:12.022198 (+ 886us) client_negotiation.cc:240] Negotiation successful
0928 03:05:12.022365 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":1,"thread_start_us":856,"threads_started":1}
I20260928 03:05:12.025617 14652 tablet_service.cc:1467] Processing CreateTablet for tablet 673ee9b4e9f843a098b2779777ee3b07 (DEFAULT_TABLE table=test-table [id=9559ce9652214b2ab6fa68661ec46da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:12.026314 14791 tablet_service.cc:1467] Processing CreateTablet for tablet 673ee9b4e9f843a098b2779777ee3b07 (DEFAULT_TABLE table=test-table [id=9559ce9652214b2ab6fa68661ec46da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:12.027403 14652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 673ee9b4e9f843a098b2779777ee3b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:12.029315 14791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 673ee9b4e9f843a098b2779777ee3b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:12.048398 15024 tablet_bootstrap.cc:492] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271: Bootstrap starting.
I20260928 03:05:12.050289 15026 tablet_bootstrap.cc:492] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0: Bootstrap starting.
I20260928 03:05:12.051472 15025 tablet_bootstrap.cc:492] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999: Bootstrap starting.
I20260928 03:05:12.054543 15024 tablet_bootstrap.cc:654] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:12.054617 15026 tablet_bootstrap.cc:654] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:12.056079 15026 log.cc:824] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0: Log is configured to *not* fsync() on all Append() calls
I20260928 03:05:12.057446 15025 tablet_bootstrap.cc:654] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:12.057942 15024 log.cc:824] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271: Log is configured to *not* fsync() on all Append() calls
I20260928 03:05:12.059136 15025 log.cc:824] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999: Log is configured to *not* fsync() on all Append() calls
I20260928 03:05:12.066807 15026 tablet_bootstrap.cc:492] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0: No bootstrap required, opened a new log
I20260928 03:05:12.067282 15026 ts_tablet_manager.cc:1397] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260928 03:05:12.074955 15025 tablet_bootstrap.cc:492] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999: No bootstrap required, opened a new log
I20260928 03:05:12.074955 15024 tablet_bootstrap.cc:492] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271: No bootstrap required, opened a new log
I20260928 03:05:12.075289 15024 ts_tablet_manager.cc:1397] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.012s
I20260928 03:05:12.075309 15025 ts_tablet_manager.cc:1397] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.014s
I20260928 03:05:12.082986 15026 raft_consensus.cc:348] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.083671 15026 raft_consensus.cc:374] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:12.083808 15026 raft_consensus.cc:729] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7518965ca94f1182344204d56b18a0, State: Initialized, Role: FOLLOWER
I20260928 03:05:12.084677 15026 consensus_queue.cc:260] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.087458 15024 raft_consensus.cc:348] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.088061 15024 raft_consensus.cc:374] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:12.088172 15024 raft_consensus.cc:729] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e08753d65844d5d85150d55bcc41271, State: Initialized, Role: FOLLOWER
I20260928 03:05:12.089000 15024 consensus_queue.cc:260] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.089952 15025 raft_consensus.cc:348] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.090579 15025 raft_consensus.cc:374] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:12.090711 15025 raft_consensus.cc:729] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8fe39b38ba14c088a1d2297d2f8c999, State: Initialized, Role: FOLLOWER
I20260928 03:05:12.091784 15025 consensus_queue.cc:260] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.098344 15024 ts_tablet_manager.cc:1428] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271: Time spent starting tablet: real 0.023s user 0.015s sys 0.007s
I20260928 03:05:12.098909 15000 heartbeater.cc:503] Master 127.6.192.62:34851 was elected leader, sending a full tablet report...
I20260928 03:05:12.099490 14861 heartbeater.cc:503] Master 127.6.192.62:34851 was elected leader, sending a full tablet report...
I20260928 03:05:12.100306 15026 ts_tablet_manager.cc:1428] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0: Time spent starting tablet: real 0.033s user 0.014s sys 0.021s
I20260928 03:05:12.102427 15025 ts_tablet_manager.cc:1428] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999: Time spent starting tablet: real 0.027s user 0.022s sys 0.004s
W20260928 03:05:12.139112 15001 tablet.cc:2369] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:05:12.283188 14723 tablet.cc:2369] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:05:12.336558 15032 raft_consensus.cc:482] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:05:12.336947 15032 raft_consensus.cc:504] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
W20260928 03:05:12.342698 14862 tablet.cc:2369] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:05:12.345378 15032 leader_election.cc:288] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e7518965ca94f1182344204d56b18a0 (127.6.192.2:46481), 7e08753d65844d5d85150d55bcc41271 (127.6.192.3:33831)
I20260928 03:05:12.362763 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:12.340160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:59083 (local address 127.6.192.2:46481)
0928 03:05:12.340396 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:12.340405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:12.352621 (+ 12216us) server_negotiation.cc:407] Connection header received
0928 03:05:12.352897 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:12.352904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:12.352986 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:12.353052 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:12.354211 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.355752 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.359271 (+ 3519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.361047 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.361206 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:12.361379 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:12.362188 (+ 809us) server_negotiation.cc:299] Negotiation successful
0928 03:05:12.362400 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 03:05:12.363664 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:12.339433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:46481 (local address 127.6.192.1:59083)
0928 03:05:12.352439 (+ 13006us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:12.352469 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:12.352679 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:12.353294 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:12.353299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:12.353327 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:12.353924 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:12.353946 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.355985 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.355995 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:12.359019 (+ 3024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:12.359043 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.361638 (+ 2595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.361647 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:12.361913 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:12.361968 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 03:05:12.362735 (+ 767us) client_negotiation.cc:240] Negotiation successful
0928 03:05:12.363030 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12795,"mutex_wait_us":30,"thread_start_us":12576,"threads_started":1}
I20260928 03:05:12.364467 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:12.339833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:60469 (local address 127.6.192.3:33831)
0928 03:05:12.340051 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:12.340060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:12.355660 (+ 15600us) server_negotiation.cc:407] Connection header received
0928 03:05:12.355981 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:12.355987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:12.356065 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:12.356129 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:12.357326 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.358513 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.361394 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.362802 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.362988 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:12.363110 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:12.363890 (+ 780us) server_negotiation.cc:299] Negotiation successful
0928 03:05:12.364078 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 03:05:12.364941 15034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:12.339944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:33831 (local address 127.6.192.1:60469)
0928 03:05:12.353961 (+ 14017us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:12.355555 (+ 1594us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:12.355788 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:12.356357 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:12.356363 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:12.356394 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:12.357083 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:12.357103 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.358727 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.358733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:12.361178 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:12.361197 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.363406 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.363413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:12.363671 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:12.363723 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 03:05:12.363838 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 03:05:12.364004 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13815,"mutex_wait_us":93,"thread_start_us":13595,"threads_started":1}
I20260928 03:05:12.365434 14811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "673ee9b4e9f843a098b2779777ee3b07" candidate_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7518965ca94f1182344204d56b18a0" is_pre_election: true
I20260928 03:05:12.366101 14811 raft_consensus.cc:2454] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8fe39b38ba14c088a1d2297d2f8c999 in term 0.
I20260928 03:05:12.366153 14950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "673ee9b4e9f843a098b2779777ee3b07" candidate_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e08753d65844d5d85150d55bcc41271" is_pre_election: true
I20260928 03:05:12.366699 14950 raft_consensus.cc:2454] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8fe39b38ba14c088a1d2297d2f8c999 in term 0.
I20260928 03:05:12.366986 14603 leader_election.cc:302] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7518965ca94f1182344204d56b18a0, a8fe39b38ba14c088a1d2297d2f8c999; no voters:
I20260928 03:05:12.367537 15032 raft_consensus.cc:2781] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:05:12.367753 15032 raft_consensus.cc:482] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:05:12.367908 15032 raft_consensus.cc:3037] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:12.370761 15032 raft_consensus.cc:504] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.371635 15032 leader_election.cc:288] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [CANDIDATE]: Term 1 election: Requested vote from peers 7e7518965ca94f1182344204d56b18a0 (127.6.192.2:46481), 7e08753d65844d5d85150d55bcc41271 (127.6.192.3:33831)
I20260928 03:05:12.372421 14950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "673ee9b4e9f843a098b2779777ee3b07" candidate_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e08753d65844d5d85150d55bcc41271"
I20260928 03:05:12.372697 14811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "673ee9b4e9f843a098b2779777ee3b07" candidate_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7518965ca94f1182344204d56b18a0"
I20260928 03:05:12.372711 14950 raft_consensus.cc:3037] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:12.372970 14811 raft_consensus.cc:3037] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:12.375577 14811 raft_consensus.cc:2454] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8fe39b38ba14c088a1d2297d2f8c999 in term 1.
I20260928 03:05:12.376173 14603 leader_election.cc:302] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7518965ca94f1182344204d56b18a0, a8fe39b38ba14c088a1d2297d2f8c999; no voters:
I20260928 03:05:12.376571 15032 raft_consensus.cc:2781] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:05:12.378093 14950 raft_consensus.cc:2454] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8fe39b38ba14c088a1d2297d2f8c999 in term 1.
I20260928 03:05:12.378306 15032 raft_consensus.cc:686] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 LEADER]: Becoming Leader. State: Replica: a8fe39b38ba14c088a1d2297d2f8c999, State: Running, Role: LEADER
I20260928 03:05:12.379311 15032 consensus_queue.cc:237] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.384748 14513 catalog_manager.cc:5495] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8fe39b38ba14c088a1d2297d2f8c999 (127.6.192.1). New cstate: current_term: 1 leader_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } health_report { overall_health: UNKNOWN } } }
I20260928 03:05:12.481074 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:12.474633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44452 (local address 127.6.192.1:46095)
0928 03:05:12.474930 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:12.474938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:12.475213 (+ 275us) server_negotiation.cc:407] Connection header received
0928 03:05:12.475551 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:12.475556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:12.475665 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:12.475847 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:05:12.476316 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.477105 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.478637 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.479092 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.479251 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:12.479786 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:05:12.480400 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:05:12.480583 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:12.480646 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 03:05:12.480737 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 03:05:12.490144 14811 raft_consensus.cc:1261] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0 [term 1 FOLLOWER]: Refusing update from remote peer a8fe39b38ba14c088a1d2297d2f8c999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:05:12.490150 14950 raft_consensus.cc:1261] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271 [term 1 FOLLOWER]: Refusing update from remote peer a8fe39b38ba14c088a1d2297d2f8c999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:05:12.491277 15035 consensus_queue.cc:1035] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:05:12.491813 15032 consensus_queue.cc:1035] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:05:12.519042 14513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:05:12.519431 14513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:05:12.523039 14513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8: Sending DeleteTablet for 3 replicas of tablet 673ee9b4e9f843a098b2779777ee3b07
I20260928 03:05:12.524603 14930 tablet_service.cc:1514] Processing DeleteTablet for tablet 673ee9b4e9f843a098b2779777ee3b07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:12 UTC) from {username='oryx', principal='oryx/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:36768
I20260928 03:05:12.524783 14652 tablet_service.cc:1514] Processing DeleteTablet for tablet 673ee9b4e9f843a098b2779777ee3b07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:12 UTC) from {username='oryx', principal='oryx/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:44444
I20260928 03:05:12.525552 15049 tablet_replica.cc:330] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271: stopping tablet replica
I20260928 03:05:12.526093 15050 tablet_replica.cc:330] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999: stopping tablet replica
I20260928 03:05:12.526018 14791 tablet_service.cc:1514] Processing DeleteTablet for tablet 673ee9b4e9f843a098b2779777ee3b07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:12 UTC) from {username='oryx', principal='oryx/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:48898
I20260928 03:05:12.526095 15049 raft_consensus.cc:2229] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:05:12.526520 15050 raft_consensus.cc:2229] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:05:12.526849 15049 raft_consensus.cc:2258] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:12.527222 15050 raft_consensus.cc:2258] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:12.528137 15051 tablet_replica.cc:330] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0: stopping tablet replica
I20260928 03:05:12.528538 15051 raft_consensus.cc:2229] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:05:12.528657 14513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 03:05:12.529040 15051 raft_consensus.cc:2258] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 03:05:12.529395 14513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:05:12.529729 15049 ts_tablet_manager.cc:1905] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:12.532799 15050 ts_tablet_manager.cc:1905] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:12.537259 15049 ts_tablet_manager.cc:1918] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:05:12.537566 15049 log.cc:1196] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/673ee9b4e9f843a098b2779777ee3b07
I20260928 03:05:12.538187 15049 ts_tablet_manager.cc:1939] T 673ee9b4e9f843a098b2779777ee3b07 P 7e08753d65844d5d85150d55bcc41271: Deleting consensus metadata
I20260928 03:05:12.539271 14930 tablet_service.cc:1467] Processing CreateTablet for tablet 6155446c5a704a4da4e77b841e6c7e84 (DEFAULT_TABLE table=test-table [id=4a959d4f4f254ca18cdb09e4dfa20d4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:12.539757 14930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6155446c5a704a4da4e77b841e6c7e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:12.540720 14499 catalog_manager.cc:4841] TS 7e08753d65844d5d85150d55bcc41271 (127.6.192.3:33831): tablet 673ee9b4e9f843a098b2779777ee3b07 (table test-table [id=9559ce9652214b2ab6fa68661ec46da5]) successfully deleted
I20260928 03:05:12.541185 15050 ts_tablet_manager.cc:1918] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:05:12.541380 15050 log.cc:1196] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/673ee9b4e9f843a098b2779777ee3b07
I20260928 03:05:12.541966 15050 ts_tablet_manager.cc:1939] T 673ee9b4e9f843a098b2779777ee3b07 P a8fe39b38ba14c088a1d2297d2f8c999: Deleting consensus metadata
I20260928 03:05:12.544201 14500 catalog_manager.cc:4841] TS a8fe39b38ba14c088a1d2297d2f8c999 (127.6.192.1:46095): tablet 673ee9b4e9f843a098b2779777ee3b07 (table test-table [id=9559ce9652214b2ab6fa68661ec46da5]) successfully deleted
I20260928 03:05:12.544399 14652 tablet_service.cc:1467] Processing CreateTablet for tablet 6155446c5a704a4da4e77b841e6c7e84 (DEFAULT_TABLE table=test-table [id=4a959d4f4f254ca18cdb09e4dfa20d4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:12.544899 14652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6155446c5a704a4da4e77b841e6c7e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:12.545405 15051 ts_tablet_manager.cc:1905] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:12.545806 15024 tablet_bootstrap.cc:492] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271: Bootstrap starting.
I20260928 03:05:12.547403 15024 tablet_bootstrap.cc:654] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:12.551851 14791 tablet_service.cc:1467] Processing CreateTablet for tablet 6155446c5a704a4da4e77b841e6c7e84 (DEFAULT_TABLE table=test-table [id=4a959d4f4f254ca18cdb09e4dfa20d4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:12.552436 14791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6155446c5a704a4da4e77b841e6c7e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:12.553448 15051 ts_tablet_manager.cc:1918] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:05:12.553807 15051 log.cc:1196] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/673ee9b4e9f843a098b2779777ee3b07
I20260928 03:05:12.554600 15051 ts_tablet_manager.cc:1939] T 673ee9b4e9f843a098b2779777ee3b07 P 7e7518965ca94f1182344204d56b18a0: Deleting consensus metadata
I20260928 03:05:12.556854 14500 catalog_manager.cc:4841] TS 7e7518965ca94f1182344204d56b18a0 (127.6.192.2:46481): tablet 673ee9b4e9f843a098b2779777ee3b07 (table test-table [id=9559ce9652214b2ab6fa68661ec46da5]) successfully deleted
I20260928 03:05:12.560184 15026 tablet_bootstrap.cc:492] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0: Bootstrap starting.
I20260928 03:05:12.560837 15025 tablet_bootstrap.cc:492] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999: Bootstrap starting.
I20260928 03:05:12.561632 15026 tablet_bootstrap.cc:654] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:12.562696 15025 tablet_bootstrap.cc:654] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:12.564340 15026 tablet_bootstrap.cc:492] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0: No bootstrap required, opened a new log
I20260928 03:05:12.564607 15026 ts_tablet_manager.cc:1397] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260928 03:05:12.565762 15026 raft_consensus.cc:348] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.566182 15026 raft_consensus.cc:374] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:12.566344 15026 raft_consensus.cc:729] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7518965ca94f1182344204d56b18a0, State: Initialized, Role: FOLLOWER
I20260928 03:05:12.566761 15026 consensus_queue.cc:260] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.567401 15024 tablet_bootstrap.cc:492] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271: No bootstrap required, opened a new log
I20260928 03:05:12.567602 15024 ts_tablet_manager.cc:1397] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.008s
I20260928 03:05:12.568239 15026 ts_tablet_manager.cc:1428] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 03:05:12.568511 15024 raft_consensus.cc:348] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.568833 15024 raft_consensus.cc:374] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:12.568936 15024 raft_consensus.cc:729] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e08753d65844d5d85150d55bcc41271, State: Initialized, Role: FOLLOWER
I20260928 03:05:12.569224 15024 consensus_queue.cc:260] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.569861 15024 ts_tablet_manager.cc:1428] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 03:05:12.572577 15025 tablet_bootstrap.cc:492] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999: No bootstrap required, opened a new log
I20260928 03:05:12.572780 15025 ts_tablet_manager.cc:1397] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260928 03:05:12.573719 15025 raft_consensus.cc:348] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.573978 15025 raft_consensus.cc:374] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:12.574093 15025 raft_consensus.cc:729] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8fe39b38ba14c088a1d2297d2f8c999, State: Initialized, Role: FOLLOWER
I20260928 03:05:12.574397 15025 consensus_queue.cc:260] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.575055 15025 ts_tablet_manager.cc:1428] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 03:05:12.582670 15031 raft_consensus.cc:482] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:05:12.582931 15031 raft_consensus.cc:504] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.585041 15031 leader_election.cc:288] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8fe39b38ba14c088a1d2297d2f8c999 (127.6.192.1:46095), 7e7518965ca94f1182344204d56b18a0 (127.6.192.2:46481)
I20260928 03:05:12.594497 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:12.586278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:40937 (local address 127.6.192.2:46481)
0928 03:05:12.586469 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:12.586476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:12.587105 (+ 629us) server_negotiation.cc:407] Connection header received
0928 03:05:12.587385 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:12.587392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:12.587506 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:12.587593 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:12.588144 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.588843 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.591844 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.593131 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.593250 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:12.593350 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:12.594048 (+ 698us) server_negotiation.cc:299] Negotiation successful
0928 03:05:12.594188 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 03:05:12.594518 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:12.585387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:55613 (local address 127.6.192.1:46095)
0928 03:05:12.585607 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:12.585614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:12.586126 (+ 512us) server_negotiation.cc:407] Connection header received
0928 03:05:12.586540 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:12.586549 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:12.586650 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:12.586713 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:12.587612 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.588960 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.591844 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.593158 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.593292 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:12.593407 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:12.594048 (+ 641us) server_negotiation.cc:299] Negotiation successful
0928 03:05:12.594197 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 03:05:12.595284 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:12.585391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:46095 (local address 127.6.192.3:55613)
0928 03:05:12.586031 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:12.586051 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:12.586262 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:12.586961 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:12.586964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:12.586986 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:12.587435 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:12.587471 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.589100 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.589107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:12.591631 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:12.591649 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.593580 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.593585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:12.593812 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:12.593888 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 03:05:12.594706 (+ 818us) client_negotiation.cc:240] Negotiation successful
0928 03:05:12.594865 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":99,"thread_start_us":305,"threads_started":1}
I20260928 03:05:12.595320 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:12.586118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:46481 (local address 127.6.192.3:40937)
0928 03:05:12.586956 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:12.586977 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:12.587221 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:12.587643 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:12.587647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:12.587671 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:12.587981 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:12.587988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.588969 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.588977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:12.591631 (+ 2654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:12.591649 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.593552 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.593565 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:12.593818 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:12.593883 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 03:05:12.594847 (+ 964us) client_negotiation.cc:240] Negotiation successful
0928 03:05:12.594975 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":261,"spinlock_wait_cycles":68480,"thread_start_us":464,"threads_started":1}
I20260928 03:05:12.596439 14811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6155446c5a704a4da4e77b841e6c7e84" candidate_uuid: "7e08753d65844d5d85150d55bcc41271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7518965ca94f1182344204d56b18a0" is_pre_election: true
I20260928 03:05:12.596369 14672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6155446c5a704a4da4e77b841e6c7e84" candidate_uuid: "7e08753d65844d5d85150d55bcc41271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" is_pre_election: true
I20260928 03:05:12.596709 14811 raft_consensus.cc:2454] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e08753d65844d5d85150d55bcc41271 in term 0.
I20260928 03:05:12.596884 14672 raft_consensus.cc:2454] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e08753d65844d5d85150d55bcc41271 in term 0.
I20260928 03:05:12.597298 14882 leader_election.cc:302] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e08753d65844d5d85150d55bcc41271, 7e7518965ca94f1182344204d56b18a0; no voters:
I20260928 03:05:12.597738 15031 raft_consensus.cc:2781] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:05:12.597896 15031 raft_consensus.cc:482] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:05:12.598018 15031 raft_consensus.cc:3037] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:12.601775 15031 raft_consensus.cc:504] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.603075 14672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6155446c5a704a4da4e77b841e6c7e84" candidate_uuid: "7e08753d65844d5d85150d55bcc41271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8fe39b38ba14c088a1d2297d2f8c999"
I20260928 03:05:12.603240 15031 leader_election.cc:288] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [CANDIDATE]: Term 1 election: Requested vote from peers a8fe39b38ba14c088a1d2297d2f8c999 (127.6.192.1:46095), 7e7518965ca94f1182344204d56b18a0 (127.6.192.2:46481)
I20260928 03:05:12.603364 14672 raft_consensus.cc:3037] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:12.603497 14811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6155446c5a704a4da4e77b841e6c7e84" candidate_uuid: "7e08753d65844d5d85150d55bcc41271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7518965ca94f1182344204d56b18a0"
I20260928 03:05:12.603760 14811 raft_consensus.cc:3037] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:12.607486 14811 raft_consensus.cc:2454] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e08753d65844d5d85150d55bcc41271 in term 1.
I20260928 03:05:12.607496 14672 raft_consensus.cc:2454] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e08753d65844d5d85150d55bcc41271 in term 1.
I20260928 03:05:12.608074 14882 leader_election.cc:302] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e08753d65844d5d85150d55bcc41271, 7e7518965ca94f1182344204d56b18a0; no voters:
I20260928 03:05:12.608551 15031 raft_consensus.cc:2781] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:05:12.609079 15031 raft_consensus.cc:686] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [term 1 LEADER]: Becoming Leader. State: Replica: 7e08753d65844d5d85150d55bcc41271, State: Running, Role: LEADER
I20260928 03:05:12.609542 15031 consensus_queue.cc:237] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.613888 14513 catalog_manager.cc:5495] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e08753d65844d5d85150d55bcc41271 (127.6.192.3). New cstate: current_term: 1 leader_uuid: "7e08753d65844d5d85150d55bcc41271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } health_report { overall_health: HEALTHY } } }
Sep 28 03:05:12 dist-test-slave-fjww krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564710, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.62@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM
I20260928 03:05:12.670323 15065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:12.654432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.6.192.62:34851)
0928 03:05:12.655388 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:12.655393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:12.655420 (+ 27us) server_negotiation.cc:407] Connection header received
0928 03:05:12.655521 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:12.655525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:12.655583 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:12.655693 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:12.656855 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.657615 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.658912 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.659319 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.659454 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:12.663383 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:12.663395 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:12.663398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:12.663428 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:12.666585 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:12.667715 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:12.667723 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:12.667728 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:12.667910 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:12.668454 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:12.668461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:12.668465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:12.668867 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:12.669060 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:12.669578 (+ 518us) server_negotiation.cc:299] Negotiation successful
0928 03:05:12.670010 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":833,"thread_start_us":684,"threads_started":1}
I20260928 03:05:12.670560 15064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:12.654371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34851 (local address 127.0.0.1:40868)
0928 03:05:12.655102 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:12.655124 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:12.655307 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:12.655720 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:12.655725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:12.656441 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:12.656647 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:12.656657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.657762 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.657767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:12.658768 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:12.658781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.659515 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.659522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:12.659724 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:12.660435 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:12.660454 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:12.663041 (+ 2587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:12.666785 (+ 3744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:12.666804 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:12.666814 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:12.667539 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:12.668049 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:12.668055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:12.668059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:12.668309 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:12.669040 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:12.669050 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:12.669265 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 03:05:12.669485 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 03:05:12.669853 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":397,"threads_started":1}
I20260928 03:05:12.677106 14513 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:40868:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 03:05:12.678593 14513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:05:12.686712 14652 tablet_service.cc:1467] Processing CreateTablet for tablet dd7e9de14d8346aaa628edff86628447 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e145c84c934458a8e32240283b54aba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 03:05:12.686849 14930 tablet_service.cc:1467] Processing CreateTablet for tablet dd7e9de14d8346aaa628edff86628447 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e145c84c934458a8e32240283b54aba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 03:05:12.687424 14652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7e9de14d8346aaa628edff86628447. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:12.687424 14930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7e9de14d8346aaa628edff86628447. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:12.692343 14791 tablet_service.cc:1467] Processing CreateTablet for tablet dd7e9de14d8346aaa628edff86628447 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e145c84c934458a8e32240283b54aba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 03:05:12.692941 14791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7e9de14d8346aaa628edff86628447. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:12.693590 15024 tablet_bootstrap.cc:492] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271: Bootstrap starting.
I20260928 03:05:12.695488 15024 tablet_bootstrap.cc:654] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:12.696164 15025 tablet_bootstrap.cc:492] T dd7e9de14d8346aaa628edff86628447 P a8fe39b38ba14c088a1d2297d2f8c999: Bootstrap starting.
I20260928 03:05:12.698521 15025 tablet_bootstrap.cc:654] T dd7e9de14d8346aaa628edff86628447 P a8fe39b38ba14c088a1d2297d2f8c999: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:12.706988 15026 tablet_bootstrap.cc:492] T dd7e9de14d8346aaa628edff86628447 P 7e7518965ca94f1182344204d56b18a0: Bootstrap starting.
I20260928 03:05:12.709821 15026 tablet_bootstrap.cc:654] T dd7e9de14d8346aaa628edff86628447 P 7e7518965ca94f1182344204d56b18a0: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:12.712113 15024 tablet_bootstrap.cc:492] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271: No bootstrap required, opened a new log
I20260928 03:05:12.712318 15024 ts_tablet_manager.cc:1397] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.004s
I20260928 03:05:12.713195 15024 raft_consensus.cc:348] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.713516 15024 raft_consensus.cc:374] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:12.713619 15024 raft_consensus.cc:729] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e08753d65844d5d85150d55bcc41271, State: Initialized, Role: FOLLOWER
I20260928 03:05:12.713963 15024 consensus_queue.cc:260] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.714711 15024 ts_tablet_manager.cc:1428] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 03:05:12.714542 15031 tablet_replica.cc:439] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } } }
I20260928 03:05:12.714920 15031 tablet_replica.cc:442] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:05:12.715327 15026 tablet_bootstrap.cc:492] T dd7e9de14d8346aaa628edff86628447 P 7e7518965ca94f1182344204d56b18a0: No bootstrap required, opened a new log
I20260928 03:05:12.715559 15026 ts_tablet_manager.cc:1397] T dd7e9de14d8346aaa628edff86628447 P 7e7518965ca94f1182344204d56b18a0: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20260928 03:05:12.716375 15026 raft_consensus.cc:348] T dd7e9de14d8346aaa628edff86628447 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.716660 15026 raft_consensus.cc:374] T dd7e9de14d8346aaa628edff86628447 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:12.716753 15026 raft_consensus.cc:729] T dd7e9de14d8346aaa628edff86628447 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7518965ca94f1182344204d56b18a0, State: Initialized, Role: FOLLOWER
I20260928 03:05:12.717044 15026 consensus_queue.cc:260] T dd7e9de14d8346aaa628edff86628447 P 7e7518965ca94f1182344204d56b18a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.717739 15026 ts_tablet_manager.cc:1428] T dd7e9de14d8346aaa628edff86628447 P 7e7518965ca94f1182344204d56b18a0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 03:05:12.717622 15030 tablet_replica.cc:439] T dd7e9de14d8346aaa628edff86628447 P 7e7518965ca94f1182344204d56b18a0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } } }
I20260928 03:05:12.718115 15030 tablet_replica.cc:442] T dd7e9de14d8346aaa628edff86628447 P 7e7518965ca94f1182344204d56b18a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:05:12.717208 15025 tablet_bootstrap.cc:492] T dd7e9de14d8346aaa628edff86628447 P a8fe39b38ba14c088a1d2297d2f8c999: No bootstrap required, opened a new log
I20260928 03:05:12.719172 15025 ts_tablet_manager.cc:1397] T dd7e9de14d8346aaa628edff86628447 P a8fe39b38ba14c088a1d2297d2f8c999: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.005s
I20260928 03:05:12.720042 15025 raft_consensus.cc:348] T dd7e9de14d8346aaa628edff86628447 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.720300 15025 raft_consensus.cc:374] T dd7e9de14d8346aaa628edff86628447 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:12.720449 15025 raft_consensus.cc:729] T dd7e9de14d8346aaa628edff86628447 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8fe39b38ba14c088a1d2297d2f8c999, State: Initialized, Role: FOLLOWER
I20260928 03:05:12.720990 15025 consensus_queue.cc:260] T dd7e9de14d8346aaa628edff86628447 P a8fe39b38ba14c088a1d2297d2f8c999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.721451 15035 tablet_replica.cc:439] T dd7e9de14d8346aaa628edff86628447 P a8fe39b38ba14c088a1d2297d2f8c999: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } } }
I20260928 03:05:12.721719 15025 ts_tablet_manager.cc:1428] T dd7e9de14d8346aaa628edff86628447 P a8fe39b38ba14c088a1d2297d2f8c999: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 03:05:12.721889 15035 tablet_replica.cc:442] T dd7e9de14d8346aaa628edff86628447 P a8fe39b38ba14c088a1d2297d2f8c999: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:05:12.881652 15031 raft_consensus.cc:482] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:05:12.881938 15031 raft_consensus.cc:504] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.882754 15031 leader_election.cc:288] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8fe39b38ba14c088a1d2297d2f8c999 (127.6.192.1:46095), 7e7518965ca94f1182344204d56b18a0 (127.6.192.2:46481)
I20260928 03:05:12.883526 14672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7e9de14d8346aaa628edff86628447" candidate_uuid: "7e08753d65844d5d85150d55bcc41271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" is_pre_election: true
I20260928 03:05:12.883684 14811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7e9de14d8346aaa628edff86628447" candidate_uuid: "7e08753d65844d5d85150d55bcc41271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7518965ca94f1182344204d56b18a0" is_pre_election: true
I20260928 03:05:12.883810 14672 raft_consensus.cc:2454] T dd7e9de14d8346aaa628edff86628447 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e08753d65844d5d85150d55bcc41271 in term 0.
I20260928 03:05:12.883980 14811 raft_consensus.cc:2454] T dd7e9de14d8346aaa628edff86628447 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e08753d65844d5d85150d55bcc41271 in term 0.
I20260928 03:05:12.884305 14882 leader_election.cc:302] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e08753d65844d5d85150d55bcc41271, a8fe39b38ba14c088a1d2297d2f8c999; no voters:
I20260928 03:05:12.884626 15031 raft_consensus.cc:2781] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:05:12.884743 15031 raft_consensus.cc:482] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:05:12.884896 15031 raft_consensus.cc:3037] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:12.888576 15031 raft_consensus.cc:504] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.889348 15031 leader_election.cc:288] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 [CANDIDATE]: Term 1 election: Requested vote from peers a8fe39b38ba14c088a1d2297d2f8c999 (127.6.192.1:46095), 7e7518965ca94f1182344204d56b18a0 (127.6.192.2:46481)
I20260928 03:05:12.890007 14811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7e9de14d8346aaa628edff86628447" candidate_uuid: "7e08753d65844d5d85150d55bcc41271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7518965ca94f1182344204d56b18a0"
I20260928 03:05:12.890026 14672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7e9de14d8346aaa628edff86628447" candidate_uuid: "7e08753d65844d5d85150d55bcc41271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8fe39b38ba14c088a1d2297d2f8c999"
I20260928 03:05:12.890277 14672 raft_consensus.cc:3037] T dd7e9de14d8346aaa628edff86628447 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:12.890300 14811 raft_consensus.cc:3037] T dd7e9de14d8346aaa628edff86628447 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:12.893597 14672 raft_consensus.cc:2454] T dd7e9de14d8346aaa628edff86628447 P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e08753d65844d5d85150d55bcc41271 in term 1.
I20260928 03:05:12.893939 14811 raft_consensus.cc:2454] T dd7e9de14d8346aaa628edff86628447 P 7e7518965ca94f1182344204d56b18a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e08753d65844d5d85150d55bcc41271 in term 1.
I20260928 03:05:12.894384 14882 leader_election.cc:302] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e08753d65844d5d85150d55bcc41271, a8fe39b38ba14c088a1d2297d2f8c999; no voters:
I20260928 03:05:12.894744 15031 raft_consensus.cc:2781] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:05:12.895166 15031 raft_consensus.cc:686] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 [term 1 LEADER]: Becoming Leader. State: Replica: 7e08753d65844d5d85150d55bcc41271, State: Running, Role: LEADER
I20260928 03:05:12.895488 15031 consensus_queue.cc:237] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:12.897244 15057 tablet_replica.cc:439] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271: TxnStatusTablet state changed. Reason: New leader 7e08753d65844d5d85150d55bcc41271. Latest consensus state: current_term: 1 leader_uuid: "7e08753d65844d5d85150d55bcc41271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } } }
I20260928 03:05:12.897670 15057 tablet_replica.cc:442] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271: This TxnStatusTablet replica's current role is: LEADER
I20260928 03:05:12.899024 15070 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 03:05:12.899287 15070 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 03:05:12.899619 14513 catalog_manager.cc:5495] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e08753d65844d5d85150d55bcc41271 (127.6.192.3). New cstate: current_term: 1 leader_uuid: "7e08753d65844d5d85150d55bcc41271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } health_report { overall_health: HEALTHY } } }
I20260928 03:05:12.958719 15064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:12.951311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:33831 (local address 127.0.0.1:36780)
0928 03:05:12.951702 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:12.951729 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:12.951910 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:12.953282 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:12.953288 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:12.953301 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 03:05:12.953522 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:12.953536 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.954854 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.954861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:12.955950 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:12.955963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.956791 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.956798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:12.957007 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:12.957073 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 03:05:12.958113 (+ 1040us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 03:05:12.958139 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 03:05:12.958232 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 03:05:12.958414 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260928 03:05:12.958756 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:12.951300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36780 (local address 127.6.192.3:33831)
0928 03:05:12.952807 (+ 1507us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:12.952814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:12.952840 (+ 26us) server_negotiation.cc:407] Connection header received
0928 03:05:12.952990 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:12.952995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:12.953093 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:12.953257 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:05:12.953686 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.954724 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.956095 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:12.956588 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:12.957226 (+ 638us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:12.957363 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:05:12.957997 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:05:12.958169 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:12.958253 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 03:05:12.958338 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1325,"thread_start_us":1090,"threads_started":1}
I20260928 03:05:12.963403 14811 raft_consensus.cc:1261] T dd7e9de14d8346aaa628edff86628447 P 7e7518965ca94f1182344204d56b18a0 [term 1 FOLLOWER]: Refusing update from remote peer 7e08753d65844d5d85150d55bcc41271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:05:12.963629 14672 raft_consensus.cc:1261] T dd7e9de14d8346aaa628edff86628447 P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 FOLLOWER]: Refusing update from remote peer 7e08753d65844d5d85150d55bcc41271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:05:12.964355 15031 consensus_queue.cc:1035] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:05:12.965006 15031 consensus_queue.cc:1035] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:05:12.972536 15035 tablet_replica.cc:439] T dd7e9de14d8346aaa628edff86628447 P a8fe39b38ba14c088a1d2297d2f8c999: TxnStatusTablet state changed. Reason: New leader 7e08753d65844d5d85150d55bcc41271. Latest consensus state: current_term: 1 leader_uuid: "7e08753d65844d5d85150d55bcc41271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } } }
I20260928 03:05:12.972925 15035 tablet_replica.cc:442] T dd7e9de14d8346aaa628edff86628447 P a8fe39b38ba14c088a1d2297d2f8c999: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:05:12.974987 15057 tablet_replica.cc:439] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e08753d65844d5d85150d55bcc41271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } } }
I20260928 03:05:12.975273 15057 tablet_replica.cc:442] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271: This TxnStatusTablet replica's current role is: LEADER
I20260928 03:05:12.977226 15040 mvcc.cc:205] Tried to move back new op lower bound from 7334153064289792000 to 7334153064022913024. Current Snapshot: MvccSnapshot[applied={T|T < 7334153064289792000 or (T in {7334153064289792000})}]
I20260928 03:05:12.984050 15039 mvcc.cc:205] Tried to move back new op lower bound from 7334153064289792000 to 7334153064022913024. Current Snapshot: MvccSnapshot[applied={T|T < 7334153064289792000}]
I20260928 03:05:12.984403 15035 tablet_replica.cc:439] T dd7e9de14d8346aaa628edff86628447 P a8fe39b38ba14c088a1d2297d2f8c999: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e08753d65844d5d85150d55bcc41271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } } }
I20260928 03:05:12.984696 15035 tablet_replica.cc:442] T dd7e9de14d8346aaa628edff86628447 P a8fe39b38ba14c088a1d2297d2f8c999: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:05:12.986158 15030 tablet_replica.cc:439] T dd7e9de14d8346aaa628edff86628447 P 7e7518965ca94f1182344204d56b18a0: TxnStatusTablet state changed. Reason: New leader 7e08753d65844d5d85150d55bcc41271. Latest consensus state: current_term: 1 leader_uuid: "7e08753d65844d5d85150d55bcc41271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } } }
I20260928 03:05:12.986452 15030 tablet_replica.cc:442] T dd7e9de14d8346aaa628edff86628447 P 7e7518965ca94f1182344204d56b18a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:05:12.986937 15031 tablet_replica.cc:439] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e08753d65844d5d85150d55bcc41271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } } }
I20260928 03:05:12.987269 15031 tablet_replica.cc:442] T dd7e9de14d8346aaa628edff86628447 P 7e08753d65844d5d85150d55bcc41271: This TxnStatusTablet replica's current role is: LEADER
I20260928 03:05:12.989836 15030 tablet_replica.cc:439] T dd7e9de14d8346aaa628edff86628447 P 7e7518965ca94f1182344204d56b18a0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e08753d65844d5d85150d55bcc41271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } } }
I20260928 03:05:12.990135 15030 tablet_replica.cc:442] T dd7e9de14d8346aaa628edff86628447 P 7e7518965ca94f1182344204d56b18a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 03:05:12.989964 15041 mvcc.cc:205] Tried to move back new op lower bound from 7334153064289792000 to 7334153064022913024. Current Snapshot: MvccSnapshot[applied={T|T < 7334153064289792000}]
I20260928 03:05:13.005571 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:13.000072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.6.192.3:33831)
0928 03:05:13.000551 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:13.000555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:13.000569 (+ 14us) server_negotiation.cc:407] Connection header received
0928 03:05:13.000625 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:13.000629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:13.000753 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:13.000864 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:05:13.001394 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.002243 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.003711 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.004187 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.004330 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:13.004745 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:05:13.004852 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:05:13.004961 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:13.005106 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 03:05:13.005192 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357}
I20260928 03:05:13.041484 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:13.033485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:55563 (local address 127.6.192.3:33831)
0928 03:05:13.033712 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:13.033718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:13.034484 (+ 766us) server_negotiation.cc:407] Connection header received
0928 03:05:13.034729 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:13.034735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:13.034846 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:13.034935 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:05:13.035631 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.036856 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.039105 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.039925 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.040061 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:13.040604 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:05:13.040740 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:05:13.040836 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:13.040997 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 03:05:13.041084 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 03:05:13.041574 15080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:13.033520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:33831 (local address 127.6.192.3:55563)
0928 03:05:13.034355 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:13.034381 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:13.034558 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:13.035075 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:13.035079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:13.035093 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 03:05:13.035465 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:13.035477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.037027 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.037032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:13.038941 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:13.038962 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.040136 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.040142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:13.040332 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:13.040416 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 03:05:13.040855 (+ 439us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 03:05:13.040866 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 03:05:13.040958 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 03:05:13.041109 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":493,"threads_started":1}
I20260928 03:05:13.053627 14811 raft_consensus.cc:1261] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0 [term 1 FOLLOWER]: Refusing update from remote peer 7e08753d65844d5d85150d55bcc41271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:05:13.054415 14672 raft_consensus.cc:1261] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 FOLLOWER]: Refusing update from remote peer 7e08753d65844d5d85150d55bcc41271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:05:13.054807 15057 consensus_queue.cc:1035] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:05:13.055123 15031 consensus_queue.cc:1035] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:05:13.145514 14514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:05:13.145812 14514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:05:13.149430 14514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8: Sending DeleteTablet for 3 replicas of tablet 6155446c5a704a4da4e77b841e6c7e84
I20260928 03:05:13.150513 14930 tablet_service.cc:1514] Processing DeleteTablet for tablet 6155446c5a704a4da4e77b841e6c7e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:13 UTC) from {username='oryx', principal='oryx/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:36768
I20260928 03:05:13.150974 14652 tablet_service.cc:1514] Processing DeleteTablet for tablet 6155446c5a704a4da4e77b841e6c7e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:13 UTC) from {username='oryx', principal='oryx/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:44444
I20260928 03:05:13.151504 14791 tablet_service.cc:1514] Processing DeleteTablet for tablet 6155446c5a704a4da4e77b841e6c7e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:13 UTC) from {username='oryx', principal='oryx/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:48898
I20260928 03:05:13.151685 15087 tablet_replica.cc:330] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999: stopping tablet replica
I20260928 03:05:13.151883 15087 raft_consensus.cc:2229] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:05:13.152096 15087 raft_consensus.cc:2258] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:13.153206 15087 ts_tablet_manager.cc:1905] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:13.156097 15088 tablet_replica.cc:330] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0: stopping tablet replica
I20260928 03:05:13.156347 15088 raft_consensus.cc:2229] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:05:13.156594 15088 raft_consensus.cc:2258] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:13.161043 15087 ts_tablet_manager.cc:1918] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 03:05:13.161216 15087 log.cc:1196] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/6155446c5a704a4da4e77b841e6c7e84
I20260928 03:05:13.161780 15087 ts_tablet_manager.cc:1939] T 6155446c5a704a4da4e77b841e6c7e84 P a8fe39b38ba14c088a1d2297d2f8c999: Deleting consensus metadata
I20260928 03:05:13.163057 14500 catalog_manager.cc:4841] TS a8fe39b38ba14c088a1d2297d2f8c999 (127.6.192.1:46095): tablet 6155446c5a704a4da4e77b841e6c7e84 (table test-table [id=4a959d4f4f254ca18cdb09e4dfa20d4c]) successfully deleted
I20260928 03:05:13.164258 15086 tablet_replica.cc:330] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271: stopping tablet replica
I20260928 03:05:13.164592 15086 raft_consensus.cc:2229] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:05:13.165431 15086 raft_consensus.cc:2258] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:13.166544 15086 ts_tablet_manager.cc:1905] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:13.167531 15088 ts_tablet_manager.cc:1905] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:13.176113 15086 ts_tablet_manager.cc:1918] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 03:05:13.176110 15088 ts_tablet_manager.cc:1918] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 03:05:13.176291 15088 log.cc:1196] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/6155446c5a704a4da4e77b841e6c7e84
I20260928 03:05:13.176303 15086 log.cc:1196] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/6155446c5a704a4da4e77b841e6c7e84
I20260928 03:05:13.176891 15088 ts_tablet_manager.cc:1939] T 6155446c5a704a4da4e77b841e6c7e84 P 7e7518965ca94f1182344204d56b18a0: Deleting consensus metadata
I20260928 03:05:13.176895 15086 ts_tablet_manager.cc:1939] T 6155446c5a704a4da4e77b841e6c7e84 P 7e08753d65844d5d85150d55bcc41271: Deleting consensus metadata
I20260928 03:05:13.178210 14499 catalog_manager.cc:4841] TS 7e08753d65844d5d85150d55bcc41271 (127.6.192.3:33831): tablet 6155446c5a704a4da4e77b841e6c7e84 (table test-table [id=4a959d4f4f254ca18cdb09e4dfa20d4c]) successfully deleted
I20260928 03:05:13.178210 14500 catalog_manager.cc:4841] TS 7e7518965ca94f1182344204d56b18a0 (127.6.192.2:46481): tablet 6155446c5a704a4da4e77b841e6c7e84 (table test-table [id=4a959d4f4f254ca18cdb09e4dfa20d4c]) successfully deleted
Sep 28 03:05:13 dist-test-slave-fjww krb5kdc[14470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 03:05:13 dist-test-slave-fjww krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:13 dist-test-slave-fjww krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 03:05:13.224551 15096 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:13.208718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.6.192.62:34851)
0928 03:05:13.209509 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:13.209517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:13.209542 (+ 25us) server_negotiation.cc:407] Connection header received
0928 03:05:13.209597 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:13.209602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:13.209665 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:13.209812 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:13.211199 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.212161 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.213423 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.213872 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.214019 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:13.224141 (+ 10122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40872: BlockingRecv error: recv got EOF from 127.0.0.1:40872 (error 108)
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":531,"threads_started":1}
Sep 28 03:05:13 dist-test-slave-fjww krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM
I20260928 03:05:13.253204 15096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:13.230193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40876 (local address 127.6.192.62:34851)
0928 03:05:13.230544 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:13.230551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:13.230819 (+ 268us) server_negotiation.cc:407] Connection header received
0928 03:05:13.231051 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:13.231057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:13.231156 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:13.231538 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:13.233055 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.234461 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.236202 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.236889 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.237300 (+ 411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:13.243780 (+ 6480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:13.243791 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:13.243796 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:13.243858 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:13.248451 (+ 4593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:13.249496 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:13.249506 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:13.249512 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:13.249738 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:13.250317 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:13.250323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:13.250327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:13.250774 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:13.251086 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:13.252501 (+ 1415us) server_negotiation.cc:299] Negotiation successful
0928 03:05:13.252774 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260928 03:05:13.258754 14514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:05:13.259752 14514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:05:13.269783 14652 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8279f785cc4205b8240d5e6eaaa736 (DEFAULT_TABLE table=test-table [id=0d447d106cd5491daaa1cd00dd378fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:13.270396 14652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b8279f785cc4205b8240d5e6eaaa736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:13.270462 14791 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8279f785cc4205b8240d5e6eaaa736 (DEFAULT_TABLE table=test-table [id=0d447d106cd5491daaa1cd00dd378fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:13.271008 14791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b8279f785cc4205b8240d5e6eaaa736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:13.277292 14930 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8279f785cc4205b8240d5e6eaaa736 (DEFAULT_TABLE table=test-table [id=0d447d106cd5491daaa1cd00dd378fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:13.277824 14930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b8279f785cc4205b8240d5e6eaaa736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:13.288005 15104 tablet_bootstrap.cc:492] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271: Bootstrap starting.
I20260928 03:05:13.289923 15103 tablet_bootstrap.cc:492] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999: Bootstrap starting.
I20260928 03:05:13.292315 15102 tablet_bootstrap.cc:492] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0: Bootstrap starting.
I20260928 03:05:13.293988 15102 tablet_bootstrap.cc:654] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:13.294781 15104 tablet_bootstrap.cc:654] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:13.295694 15103 tablet_bootstrap.cc:654] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:13.298615 15102 tablet_bootstrap.cc:492] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0: No bootstrap required, opened a new log
I20260928 03:05:13.298861 15102 ts_tablet_manager.cc:1397] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 03:05:13.298955 15104 tablet_bootstrap.cc:492] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271: No bootstrap required, opened a new log
I20260928 03:05:13.299149 15104 ts_tablet_manager.cc:1397] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260928 03:05:13.300156 15102 raft_consensus.cc:348] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:13.300380 15104 raft_consensus.cc:348] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:13.300709 15104 raft_consensus.cc:374] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:13.300624 15102 raft_consensus.cc:374] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:13.300855 15104 raft_consensus.cc:729] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e08753d65844d5d85150d55bcc41271, State: Initialized, Role: FOLLOWER
I20260928 03:05:13.301025 15102 raft_consensus.cc:729] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7518965ca94f1182344204d56b18a0, State: Initialized, Role: FOLLOWER
I20260928 03:05:13.301219 15104 consensus_queue.cc:260] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:13.301419 15102 consensus_queue.cc:260] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:13.301991 15104 ts_tablet_manager.cc:1428] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 03:05:13.303079 15102 ts_tablet_manager.cc:1428] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 03:05:13.313088 15103 tablet_bootstrap.cc:492] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999: No bootstrap required, opened a new log
I20260928 03:05:13.313311 15103 ts_tablet_manager.cc:1397] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.003s
I20260928 03:05:13.314260 15103 raft_consensus.cc:348] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:13.314571 15103 raft_consensus.cc:374] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:13.314668 15103 raft_consensus.cc:729] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8fe39b38ba14c088a1d2297d2f8c999, State: Initialized, Role: FOLLOWER
I20260928 03:05:13.314947 15103 consensus_queue.cc:260] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:13.315683 15103 ts_tablet_manager.cc:1428] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 03:05:13.323740 15031 raft_consensus.cc:482] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:05:13.323985 15031 raft_consensus.cc:504] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:13.324761 15031 leader_election.cc:288] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8fe39b38ba14c088a1d2297d2f8c999 (127.6.192.1:46095), 7e7518965ca94f1182344204d56b18a0 (127.6.192.2:46481)
I20260928 03:05:13.325328 14671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8279f785cc4205b8240d5e6eaaa736" candidate_uuid: "7e08753d65844d5d85150d55bcc41271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" is_pre_election: true
I20260928 03:05:13.325438 14811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8279f785cc4205b8240d5e6eaaa736" candidate_uuid: "7e08753d65844d5d85150d55bcc41271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7518965ca94f1182344204d56b18a0" is_pre_election: true
I20260928 03:05:13.325584 14671 raft_consensus.cc:2454] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e08753d65844d5d85150d55bcc41271 in term 0.
I20260928 03:05:13.325739 14811 raft_consensus.cc:2454] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e08753d65844d5d85150d55bcc41271 in term 0.
I20260928 03:05:13.326033 14882 leader_election.cc:302] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e08753d65844d5d85150d55bcc41271, a8fe39b38ba14c088a1d2297d2f8c999; no voters:
I20260928 03:05:13.326330 15031 raft_consensus.cc:2781] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:05:13.326442 15031 raft_consensus.cc:482] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:05:13.326550 15031 raft_consensus.cc:3037] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:13.328984 15031 raft_consensus.cc:504] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:13.329583 15031 leader_election.cc:288] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [CANDIDATE]: Term 1 election: Requested vote from peers a8fe39b38ba14c088a1d2297d2f8c999 (127.6.192.1:46095), 7e7518965ca94f1182344204d56b18a0 (127.6.192.2:46481)
I20260928 03:05:13.330102 14671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8279f785cc4205b8240d5e6eaaa736" candidate_uuid: "7e08753d65844d5d85150d55bcc41271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8fe39b38ba14c088a1d2297d2f8c999"
I20260928 03:05:13.330312 14671 raft_consensus.cc:3037] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:13.330231 14811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8279f785cc4205b8240d5e6eaaa736" candidate_uuid: "7e08753d65844d5d85150d55bcc41271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7518965ca94f1182344204d56b18a0"
I20260928 03:05:13.330504 14811 raft_consensus.cc:3037] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:13.332913 14671 raft_consensus.cc:2454] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e08753d65844d5d85150d55bcc41271 in term 1.
I20260928 03:05:13.333519 14882 leader_election.cc:302] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e08753d65844d5d85150d55bcc41271, a8fe39b38ba14c088a1d2297d2f8c999; no voters:
I20260928 03:05:13.333897 14811 raft_consensus.cc:2454] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e08753d65844d5d85150d55bcc41271 in term 1.
I20260928 03:05:13.334192 15031 raft_consensus.cc:2781] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:05:13.334422 15031 raft_consensus.cc:686] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [term 1 LEADER]: Becoming Leader. State: Replica: 7e08753d65844d5d85150d55bcc41271, State: Running, Role: LEADER
I20260928 03:05:13.334707 15031 consensus_queue.cc:237] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:13.337291 14515 catalog_manager.cc:5495] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e08753d65844d5d85150d55bcc41271 (127.6.192.3). New cstate: current_term: 1 leader_uuid: "7e08753d65844d5d85150d55bcc41271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } health_report { overall_health: HEALTHY } } }
I20260928 03:05:13.361156 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:13.354132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.6.192.3:33831)
0928 03:05:13.354371 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:13.354379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:13.354492 (+ 113us) server_negotiation.cc:407] Connection header received
0928 03:05:13.354700 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:13.354707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:13.354819 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:13.354948 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:05:13.355357 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.356602 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.358678 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.359548 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.359710 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:13.360223 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:05:13.360419 (+ 196us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:05:13.360533 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:13.360697 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 03:05:13.360794 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 03:05:13.372756 14811 raft_consensus.cc:1261] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0 [term 1 FOLLOWER]: Refusing update from remote peer 7e08753d65844d5d85150d55bcc41271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:05:13.373982 15081 consensus_queue.cc:1035] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:05:13.375432 14671 raft_consensus.cc:1261] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 FOLLOWER]: Refusing update from remote peer 7e08753d65844d5d85150d55bcc41271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:05:13.376178 15081 consensus_queue.cc:1035] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:05:13.402457 14513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:05:13.402737 14513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:05:13.407310 14513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8: Sending DeleteTablet for 3 replicas of tablet 2b8279f785cc4205b8240d5e6eaaa736
I20260928 03:05:13.408615 14930 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b8279f785cc4205b8240d5e6eaaa736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:13 UTC) from {username='oryx', principal='oryx/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:36768
I20260928 03:05:13.409194 15086 tablet_replica.cc:330] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271: stopping tablet replica
I20260928 03:05:13.409353 14652 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b8279f785cc4205b8240d5e6eaaa736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:13 UTC) from {username='oryx', principal='oryx/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:44444
I20260928 03:05:13.409420 15086 raft_consensus.cc:2229] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:05:13.409785 15086 raft_consensus.cc:2258] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:13.409754 14791 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b8279f785cc4205b8240d5e6eaaa736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:13 UTC) from {username='oryx', principal='oryx/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:48898
I20260928 03:05:13.410104 15087 tablet_replica.cc:330] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999: stopping tablet replica
I20260928 03:05:13.410177 14790 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b8279f785cc4205b8240d5e6eaaa736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:13 UTC) from {username='oryx', principal='oryx/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:48898
I20260928 03:05:13.410348 15087 raft_consensus.cc:2229] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:05:13.410630 15087 raft_consensus.cc:2258] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:13.410959 15088 tablet_replica.cc:330] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0: stopping tablet replica
I20260928 03:05:13.410856 14513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 03:05:13.411211 15088 raft_consensus.cc:2229] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:05:13.411504 15088 raft_consensus.cc:2258] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 03:05:13.412227 14513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:05:13.413178 15087 ts_tablet_manager.cc:1905] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:13.420655 15087 ts_tablet_manager.cc:1918] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:05:13.420863 15087 log.cc:1196] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/2b8279f785cc4205b8240d5e6eaaa736
I20260928 03:05:13.421370 15087 ts_tablet_manager.cc:1939] T 2b8279f785cc4205b8240d5e6eaaa736 P a8fe39b38ba14c088a1d2297d2f8c999: Deleting consensus metadata
I20260928 03:05:13.423938 15088 ts_tablet_manager.cc:1905] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:13.426779 14930 tablet_service.cc:1467] Processing CreateTablet for tablet c31b7e39498a407db9ccb88ce1459acc (DEFAULT_TABLE table=test-table [id=029e422c5d22404ab218a923374542c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:13.426893 14652 tablet_service.cc:1467] Processing CreateTablet for tablet c31b7e39498a407db9ccb88ce1459acc (DEFAULT_TABLE table=test-table [id=029e422c5d22404ab218a923374542c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:13.427521 14652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31b7e39498a407db9ccb88ce1459acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:13.427570 14930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31b7e39498a407db9ccb88ce1459acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:13.431248 15088 ts_tablet_manager.cc:1918] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:05:13.431478 15088 log.cc:1196] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/2b8279f785cc4205b8240d5e6eaaa736
I20260928 03:05:13.431995 15088 ts_tablet_manager.cc:1939] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e7518965ca94f1182344204d56b18a0: Deleting consensus metadata
I20260928 03:05:13.432379 14790 tablet_service.cc:1467] Processing CreateTablet for tablet c31b7e39498a407db9ccb88ce1459acc (DEFAULT_TABLE table=test-table [id=029e422c5d22404ab218a923374542c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:13.432919 14790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31b7e39498a407db9ccb88ce1459acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:13.433534 14500 catalog_manager.cc:4841] TS 7e7518965ca94f1182344204d56b18a0 (127.6.192.2:46481): tablet 2b8279f785cc4205b8240d5e6eaaa736 (table test-table [id=0d447d106cd5491daaa1cd00dd378fbd]) successfully deleted
I20260928 03:05:13.434886 14500 catalog_manager.cc:4841] TS a8fe39b38ba14c088a1d2297d2f8c999 (127.6.192.1:46095): tablet 2b8279f785cc4205b8240d5e6eaaa736 (table test-table [id=0d447d106cd5491daaa1cd00dd378fbd]) successfully deleted
W20260928 03:05:13.435633 14500 catalog_manager.cc:4808] TS 7e7518965ca94f1182344204d56b18a0 (127.6.192.2:46481): delete failed for tablet 2b8279f785cc4205b8240d5e6eaaa736 because the tablet was not found. No further retry: Not found: Tablet not found: 2b8279f785cc4205b8240d5e6eaaa736
I20260928 03:05:13.437084 15103 tablet_bootstrap.cc:492] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999: Bootstrap starting.
I20260928 03:05:13.439128 15103 tablet_bootstrap.cc:654] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:13.441797 15086 ts_tablet_manager.cc:1905] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:13.442036 15103 tablet_bootstrap.cc:492] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999: No bootstrap required, opened a new log
I20260928 03:05:13.442246 15103 ts_tablet_manager.cc:1397] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260928 03:05:13.443182 15103 raft_consensus.cc:348] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:13.443624 15103 raft_consensus.cc:374] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:13.443723 15103 raft_consensus.cc:729] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8fe39b38ba14c088a1d2297d2f8c999, State: Initialized, Role: FOLLOWER
I20260928 03:05:13.444070 15103 consensus_queue.cc:260] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:13.446087 15103 ts_tablet_manager.cc:1428] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 03:05:13.451715 15086 ts_tablet_manager.cc:1918] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:05:13.451858 15104 tablet_bootstrap.cc:492] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271: Bootstrap starting.
I20260928 03:05:13.452266 15102 tablet_bootstrap.cc:492] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0: Bootstrap starting.
I20260928 03:05:13.453676 15104 tablet_bootstrap.cc:654] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:13.454159 15102 tablet_bootstrap.cc:654] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:13.456535 15102 tablet_bootstrap.cc:492] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0: No bootstrap required, opened a new log
I20260928 03:05:13.456723 15102 ts_tablet_manager.cc:1397] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 03:05:13.457592 15102 raft_consensus.cc:348] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:13.457878 15102 raft_consensus.cc:374] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:13.457981 15102 raft_consensus.cc:729] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7518965ca94f1182344204d56b18a0, State: Initialized, Role: FOLLOWER
I20260928 03:05:13.458261 15102 consensus_queue.cc:260] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:13.458675 15086 log.cc:1196] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/2b8279f785cc4205b8240d5e6eaaa736
I20260928 03:05:13.459031 15102 ts_tablet_manager.cc:1428] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 03:05:13.459204 15086 ts_tablet_manager.cc:1939] T 2b8279f785cc4205b8240d5e6eaaa736 P 7e08753d65844d5d85150d55bcc41271: Deleting consensus metadata
I20260928 03:05:13.461772 14499 catalog_manager.cc:4841] TS 7e08753d65844d5d85150d55bcc41271 (127.6.192.3:33831): tablet 2b8279f785cc4205b8240d5e6eaaa736 (table test-table [id=0d447d106cd5491daaa1cd00dd378fbd]) successfully deleted
I20260928 03:05:13.466017 15104 tablet_bootstrap.cc:492] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271: No bootstrap required, opened a new log
I20260928 03:05:13.466217 15104 ts_tablet_manager.cc:1397] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.002s
I20260928 03:05:13.466950 15104 raft_consensus.cc:348] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:13.467271 15104 raft_consensus.cc:374] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:13.467370 15104 raft_consensus.cc:729] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e08753d65844d5d85150d55bcc41271, State: Initialized, Role: FOLLOWER
I20260928 03:05:13.467717 15104 consensus_queue.cc:260] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:13.468526 15104 ts_tablet_manager.cc:1428] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 03:05:13.599584 15035 raft_consensus.cc:482] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:05:13.599829 15035 raft_consensus.cc:504] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:13.600698 15035 leader_election.cc:288] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e7518965ca94f1182344204d56b18a0 (127.6.192.2:46481), 7e08753d65844d5d85150d55bcc41271 (127.6.192.3:33831)
I20260928 03:05:13.601321 14811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31b7e39498a407db9ccb88ce1459acc" candidate_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7518965ca94f1182344204d56b18a0" is_pre_election: true
I20260928 03:05:13.601384 14950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31b7e39498a407db9ccb88ce1459acc" candidate_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e08753d65844d5d85150d55bcc41271" is_pre_election: true
I20260928 03:05:13.601583 14811 raft_consensus.cc:2454] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8fe39b38ba14c088a1d2297d2f8c999 in term 0.
I20260928 03:05:13.601748 14950 raft_consensus.cc:2454] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8fe39b38ba14c088a1d2297d2f8c999 in term 0.
I20260928 03:05:13.602128 14603 leader_election.cc:302] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7518965ca94f1182344204d56b18a0, a8fe39b38ba14c088a1d2297d2f8c999; no voters:
I20260928 03:05:13.602449 15035 raft_consensus.cc:2781] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:05:13.602553 15035 raft_consensus.cc:482] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:05:13.602689 15035 raft_consensus.cc:3037] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:13.604910 15035 raft_consensus.cc:504] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:13.605572 15035 leader_election.cc:288] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [CANDIDATE]: Term 1 election: Requested vote from peers 7e7518965ca94f1182344204d56b18a0 (127.6.192.2:46481), 7e08753d65844d5d85150d55bcc41271 (127.6.192.3:33831)
I20260928 03:05:13.606117 14811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31b7e39498a407db9ccb88ce1459acc" candidate_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7518965ca94f1182344204d56b18a0"
I20260928 03:05:13.606230 14950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31b7e39498a407db9ccb88ce1459acc" candidate_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e08753d65844d5d85150d55bcc41271"
I20260928 03:05:13.606367 14811 raft_consensus.cc:3037] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:13.606474 14950 raft_consensus.cc:3037] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:13.609613 14811 raft_consensus.cc:2454] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8fe39b38ba14c088a1d2297d2f8c999 in term 1.
I20260928 03:05:13.609622 14950 raft_consensus.cc:2454] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8fe39b38ba14c088a1d2297d2f8c999 in term 1.
I20260928 03:05:13.610128 14602 leader_election.cc:302] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e08753d65844d5d85150d55bcc41271, a8fe39b38ba14c088a1d2297d2f8c999; no voters:
I20260928 03:05:13.610531 15035 raft_consensus.cc:2781] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:05:13.610982 15035 raft_consensus.cc:686] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 LEADER]: Becoming Leader. State: Replica: a8fe39b38ba14c088a1d2297d2f8c999, State: Running, Role: LEADER
I20260928 03:05:13.611397 15035 consensus_queue.cc:237] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } }
I20260928 03:05:13.614948 14514 catalog_manager.cc:5495] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8fe39b38ba14c088a1d2297d2f8c999 (127.6.192.1). New cstate: current_term: 1 leader_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe39b38ba14c088a1d2297d2f8c999" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 46095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 } health_report { overall_health: UNKNOWN } } }
I20260928 03:05:13.677505 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:13.669002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44468 (local address 127.6.192.1:46095)
0928 03:05:13.671928 (+ 2926us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:13.671936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:13.671964 (+ 28us) server_negotiation.cc:407] Connection header received
0928 03:05:13.672050 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:13.672055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:13.672143 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:13.672269 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:05:13.672775 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.673980 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.675337 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.676036 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.676181 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:13.676628 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:05:13.676775 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:05:13.676869 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:13.677002 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 03:05:13.677084 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2784,"thread_start_us":2619,"threads_started":1}
I20260928 03:05:13.694023 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:13.688195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:50477 (local address 127.6.192.3:33831)
0928 03:05:13.688359 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:13.688363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:13.688932 (+ 569us) server_negotiation.cc:407] Connection header received
0928 03:05:13.689155 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:13.689159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:13.689229 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:13.689320 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:05:13.689805 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.690635 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.692177 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.692647 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.692792 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:13.693218 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:05:13.693335 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:05:13.693458 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:13.693599 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 03:05:13.693685 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 03:05:13.694104 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:13.688185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:33831 (local address 127.6.192.1:50477)
0928 03:05:13.688821 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:13.688837 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:13.689014 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:13.689353 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:13.689357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:13.689370 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 03:05:13.689665 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:13.689673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.690797 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.690802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:13.692037 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:13.692050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.692831 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.692840 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:13.692981 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:13.693041 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 03:05:13.693471 (+ 430us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 03:05:13.693489 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 03:05:13.693575 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 03:05:13.693726 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":345,"threads_started":1}
I20260928 03:05:13.703794 14950 raft_consensus.cc:1261] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271 [term 1 FOLLOWER]: Refusing update from remote peer a8fe39b38ba14c088a1d2297d2f8c999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:05:13.703927 14811 raft_consensus.cc:1261] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0 [term 1 FOLLOWER]: Refusing update from remote peer a8fe39b38ba14c088a1d2297d2f8c999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:05:13.704434 15115 consensus_queue.cc:1035] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e08753d65844d5d85150d55bcc41271" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:05:13.704913 15035 consensus_queue.cc:1035] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7518965ca94f1182344204d56b18a0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:05:13.749680 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:13.741182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:58757 (local address 127.6.192.1:46095)
0928 03:05:13.741367 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:13.741374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:13.741557 (+ 183us) server_negotiation.cc:407] Connection header received
0928 03:05:13.741778 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:13.741797 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:13.741885 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:13.741988 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:05:13.742949 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.744119 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.746527 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.747268 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.747412 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:13.748893 (+ 1481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:05:13.749034 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:05:13.749152 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:13.749294 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 03:05:13.749383 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 03:05:13.749974 15126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:13.740669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:46095 (local address 127.6.192.3:58757)
0928 03:05:13.741446 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:13.741466 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:13.741619 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:13.742018 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:13.742023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:13.742036 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 03:05:13.742369 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:13.742385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.744340 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.744352 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:13.746348 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:13.746372 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.747582 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.748358 (+ 776us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:13.748564 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:13.748620 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 03:05:13.749160 (+ 540us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 03:05:13.749176 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 03:05:13.749268 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 03:05:13.749439 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":392,"threads_started":1}
I20260928 03:05:13.790925 14514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:05:13.791256 14514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:05:13.794512 14514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 924cd42c531346eaa908271ad04ca8a8: Sending DeleteTablet for 3 replicas of tablet c31b7e39498a407db9ccb88ce1459acc
I20260928 03:05:13.795544 14790 tablet_service.cc:1514] Processing DeleteTablet for tablet c31b7e39498a407db9ccb88ce1459acc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:13 UTC) from {username='oryx', principal='oryx/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:48898
I20260928 03:05:13.795760 14930 tablet_service.cc:1514] Processing DeleteTablet for tablet c31b7e39498a407db9ccb88ce1459acc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:13 UTC) from {username='oryx', principal='oryx/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:36768
I20260928 03:05:13.795917 15088 tablet_replica.cc:330] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0: stopping tablet replica
I20260928 03:05:13.796090 15086 tablet_replica.cc:330] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271: stopping tablet replica
I20260928 03:05:13.796097 15088 raft_consensus.cc:2229] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:05:13.796238 15086 raft_consensus.cc:2229] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:05:13.796314 15088 raft_consensus.cc:2258] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:13.796407 15086 raft_consensus.cc:2258] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:13.796694 14652 tablet_service.cc:1514] Processing DeleteTablet for tablet c31b7e39498a407db9ccb88ce1459acc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:13 UTC) from {username='oryx', principal='oryx/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:44444
I20260928 03:05:13.797005 15087 tablet_replica.cc:330] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999: stopping tablet replica
I20260928 03:05:13.797206 15087 raft_consensus.cc:2229] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:05:13.797559 15088 ts_tablet_manager.cc:1905] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:13.797617 15087 raft_consensus.cc:2258] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:13.798179 15086 ts_tablet_manager.cc:1905] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:13.800523 15087 ts_tablet_manager.cc:1905] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:13.804731 15088 ts_tablet_manager.cc:1918] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 03:05:13.804956 15088 log.cc:1196] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/c31b7e39498a407db9ccb88ce1459acc
I20260928 03:05:13.805363 15086 ts_tablet_manager.cc:1918] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 03:05:13.805508 15088 ts_tablet_manager.cc:1939] T c31b7e39498a407db9ccb88ce1459acc P 7e7518965ca94f1182344204d56b18a0: Deleting consensus metadata
I20260928 03:05:13.805564 15086 log.cc:1196] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/c31b7e39498a407db9ccb88ce1459acc
I20260928 03:05:13.806885 14500 catalog_manager.cc:4841] TS 7e7518965ca94f1182344204d56b18a0 (127.6.192.2:46481): tablet c31b7e39498a407db9ccb88ce1459acc (table test-table [id=029e422c5d22404ab218a923374542c2]) successfully deleted
I20260928 03:05:13.808681 15087 ts_tablet_manager.cc:1918] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 03:05:13.808892 15087 log.cc:1196] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/c31b7e39498a407db9ccb88ce1459acc
I20260928 03:05:13.809372 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 14586
I20260928 03:05:13.809520 15087 ts_tablet_manager.cc:1939] T c31b7e39498a407db9ccb88ce1459acc P a8fe39b38ba14c088a1d2297d2f8c999: Deleting consensus metadata
I20260928 03:05:13.811038 14500 catalog_manager.cc:4841] TS a8fe39b38ba14c088a1d2297d2f8c999 (127.6.192.1:46095): tablet c31b7e39498a407db9ccb88ce1459acc (table test-table [id=029e422c5d22404ab218a923374542c2]) successfully deleted
I20260928 03:05:13.816497 15086 ts_tablet_manager.cc:1939] T c31b7e39498a407db9ccb88ce1459acc P 7e08753d65844d5d85150d55bcc41271: Deleting consensus metadata
I20260928 03:05:13.817708 14499 catalog_manager.cc:4841] TS 7e08753d65844d5d85150d55bcc41271 (127.6.192.3:33831): tablet c31b7e39498a407db9ccb88ce1459acc (table test-table [id=029e422c5d22404ab218a923374542c2]) successfully deleted
Sep 28 03:05:13 dist-test-slave-fjww krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.192.1@KRBTEST.COM
I20260928 03:05:13.838094 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:13.810666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44482 (local address 127.6.192.1:46095)
0928 03:05:13.810865 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:13.810871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:13.816659 (+ 5788us) server_negotiation.cc:407] Connection header received
0928 03:05:13.816947 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:13.816955 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:13.817066 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:13.817182 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:13.819503 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.820820 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.823041 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:13.823805 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:13.823938 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:13.829607 (+ 5669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:13.829629 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:13.829655 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:13.829714 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:13.834464 (+ 4750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:13.835416 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:13.835427 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:13.835431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:13.835734 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:13.836163 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:13.836169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:13.836173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:13.836682 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:13.837013 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:13.837364 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 03:05:13.837620 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 03:05:13.838934 14717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:13.970414 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 14586
I20260928 03:05:13.996698 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 14726
Sep 28 03:05:14 dist-test-slave-fjww krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.192.2@KRBTEST.COM
I20260928 03:05:14.016149 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:13.997636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48904 (local address 127.6.192.2:46481)
0928 03:05:13.999937 (+ 2301us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:13.999942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:13.999960 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:05:14.000018 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:14.000021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:14.000087 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:14.000229 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:14.001637 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:14.002486 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:14.003984 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:14.004471 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:14.004612 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:14.009367 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:14.009384 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:14.009400 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:14.009442 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:14.012926 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:14.013754 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:14.013764 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:14.013767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:14.013931 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:14.014354 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:14.014360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:14.014364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:14.014767 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:14.015105 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:14.015515 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 03:05:14.015787 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2196,"thread_start_us":2061,"threads_started":1}
I20260928 03:05:14.016880 14856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:14.118975 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 14726
I20260928 03:05:14.146353 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 14865
Sep 28 03:05:14 dist-test-slave-fjww krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.192.3@KRBTEST.COM
I20260928 03:05:14.162812 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:14.147290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36798 (local address 127.6.192.3:33831)
0928 03:05:14.147506 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:14.147511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:14.147532 (+ 21us) server_negotiation.cc:407] Connection header received
0928 03:05:14.147646 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:14.147650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:14.147714 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:14.147860 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:14.149314 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:14.150011 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:14.151260 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:14.151839 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:14.151976 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:14.156756 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:14.156773 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:14.156790 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:14.156830 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:14.159887 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:14.160616 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:14.160624 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:14.160626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:14.160774 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:14.161153 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:14.161159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:14.161163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:14.161566 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:14.161882 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:14.162190 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 03:05:14.162433 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 03:05:14.163755 14995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:14.317453 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 14865
I20260928 03:05:14.342681 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 14483
I20260928 03:05:14.344290 14555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:14.465366 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 14483
2026-09-28T03:05:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6240 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:05:14 dist-test-slave-fjww krb5kdc[15137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:05:14 dist-test-slave-fjww krb5kdc[15137](info): set up 2 sockets
Sep 28 03:05:14 dist-test-slave-fjww krb5kdc[15137](info): commencing operation
krb5kdc: starting...
W20260928 03:05:15.640758 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:05:15 dist-test-slave-fjww krb5kdc[15137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:05:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:05:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:05:15.887677 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:32921
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:32921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:05:16.229270 15150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:16.230026 15150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:16.230207 15150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:16.239508 15150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:05:16.239683 15150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:16.239778 15150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:16.239881 15150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:05:16.239981 15150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:05:16.251788 15150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:32921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:32921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:16.253333 15150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:16.254873 15150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:16.267885 15156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:16.267874 15158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:16.267920 15155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:16.268978 15150 server_base.cc:1034] running on GCE node
I20260928 03:05:16.269893 15150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:16.271574 15150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:16.272862 15150 hybrid_clock.cc:648] HybridClock initialized: now 1790564716272798 us; error 64 us; skew 500 ppm
I20260928 03:05:16.273272 15150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T03:05:16Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1880 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:05:16 dist-test-slave-fjww krb5kdc[15164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:05:16 dist-test-slave-fjww krb5kdc[15164](info): set up 2 sockets
Sep 28 03:05:16 dist-test-slave-fjww krb5kdc[15164](info): commencing operation
krb5kdc: starting...
W20260928 03:05:17.480600 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:05:17 dist-test-slave-fjww krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:05:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:05:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:05:17.723686 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:41481
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:41481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:05:18.070607 15177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:18.070919 15177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:18.071081 15177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:18.078655 15177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:05:18.078778 15177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:18.078878 15177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:18.078970 15177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:05:18.079047 15177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:05:18.089318 15177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:41481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:41481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:18.090670 15177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:18.092649 15177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:18.107527 15185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:18.107846 15182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:18.107465 15183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:18.108186 15177 server_base.cc:1034] running on GCE node
I20260928 03:05:18.109313 15177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:18.111003 15177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:18.112282 15177 hybrid_clock.cc:648] HybridClock initialized: now 1790564718112162 us; error 112 us; skew 500 ppm
I20260928 03:05:18.112843 15177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:18 dist-test-slave-fjww krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564718, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:18.121308 15177 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:05:18.123093 15177 webserver.cc:460] Webserver started at http://127.6.192.62:42825/ using document root <none> and password file <none>
I20260928 03:05:18.123930 15177 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:18.124081 15177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:18.124418 15177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:18.127791 15177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "0841e77eb42946fd975f3e72c0a51c0c"
format_stamp: "Formatted at 2026-09-28 03:05:18 on dist-test-slave-fjww"
server_key: "bf11e8f940c33fbd99e716603da9a6cf"
server_key_iv: "b632de68c96496c47d8688cbf987cd5e"
server_key_version: "encryptionkey@0"
I20260928 03:05:18.128677 15177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "0841e77eb42946fd975f3e72c0a51c0c"
format_stamp: "Formatted at 2026-09-28 03:05:18 on dist-test-slave-fjww"
server_key: "bf11e8f940c33fbd99e716603da9a6cf"
server_key_iv: "b632de68c96496c47d8688cbf987cd5e"
server_key_version: "encryptionkey@0"
I20260928 03:05:18.134307 15177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:05:18.138345 15192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:18.139850 15177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:05:18.140048 15177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "0841e77eb42946fd975f3e72c0a51c0c"
format_stamp: "Formatted at 2026-09-28 03:05:18 on dist-test-slave-fjww"
server_key: "bf11e8f940c33fbd99e716603da9a6cf"
server_key_iv: "b632de68c96496c47d8688cbf987cd5e"
server_key_version: "encryptionkey@0"
I20260928 03:05:18.140268 15177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:18.163345 15177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:18.168009 15177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:18.168308 15177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:18.187686 15177 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:41481
I20260928 03:05:18.187681 15243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:41481 every 8 connection(s)
I20260928 03:05:18.189143 15177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:05:18.193101 15244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:18.198158 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 15177
I20260928 03:05:18.198498 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:05:18.199159 6912 external_mini_cluster.cc:1428] Setting key 953bc2d36ae91597b3cd3c4a17838ce5
I20260928 03:05:18.206404 15244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c: Bootstrap starting.
I20260928 03:05:18.210891 15244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:18.212572 15244 log.cc:824] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c: Log is configured to *not* fsync() on all Append() calls
Sep 28 03:05:18 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:05:18.216634 15244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c: No bootstrap required, opened a new log
I20260928 03:05:18.225394 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:18.201402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45276 (local address 127.6.192.62:41481)
0928 03:05:18.202269 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:18.202281 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:18.202328 (+ 47us) server_negotiation.cc:407] Connection header received
0928 03:05:18.204190 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:18.204215 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:18.204656 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:18.204989 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:18.207261 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:18.208810 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:18.211045 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:18.211990 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:18.212156 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:18.217451 (+ 5295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:18.217462 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:18.217478 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:18.217525 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:18.220985 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:18.221783 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:18.221791 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:18.221798 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:18.221994 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:18.222403 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:18.222409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:18.222412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:18.223005 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:18.223337 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:18.223845 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 03:05:18.224289 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":418,"threads_started":1}
I20260928 03:05:18.226758 15244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0841e77eb42946fd975f3e72c0a51c0c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41481 } }
I20260928 03:05:18.227249 15244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:18.227403 15244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0841e77eb42946fd975f3e72c0a51c0c, State: Initialized, Role: FOLLOWER
I20260928 03:05:18.228174 15244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0841e77eb42946fd975f3e72c0a51c0c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41481 } }
I20260928 03:05:18.228541 15244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:05:18.228717 15244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:05:18.228909 15244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:18.231666 15244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0841e77eb42946fd975f3e72c0a51c0c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41481 } }
I20260928 03:05:18.232359 15244 leader_election.cc:302] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0841e77eb42946fd975f3e72c0a51c0c; no voters:
I20260928 03:05:18.233242 15244 leader_election.cc:288] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:05:18.233408 15249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:05:18.234258 15249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [term 1 LEADER]: Becoming Leader. State: Replica: 0841e77eb42946fd975f3e72c0a51c0c, State: Running, Role: LEADER
I20260928 03:05:18.235085 15249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0841e77eb42946fd975f3e72c0a51c0c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41481 } }
I20260928 03:05:18.236126 15244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:05:18.238627 15250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0841e77eb42946fd975f3e72c0a51c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0841e77eb42946fd975f3e72c0a51c0c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41481 } } }
I20260928 03:05:18.239024 15250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [sys.catalog]: This master's current role is: LEADER
I20260928 03:05:18.239434 15251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0841e77eb42946fd975f3e72c0a51c0c. Latest consensus state: current_term: 1 leader_uuid: "0841e77eb42946fd975f3e72c0a51c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0841e77eb42946fd975f3e72c0a51c0c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41481 } } }
I20260928 03:05:18.239853 15251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [sys.catalog]: This master's current role is: LEADER
I20260928 03:05:18.240716 15255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:05:18.247005 15255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:05:18.261303 15255 catalog_manager.cc:1347] Generated new cluster ID: 5194ce479e13435293817595c7c7cff1
I20260928 03:05:18.261464 15255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:05:18.280076 15255 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:05:18.281476 15255 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:05:18.295179 15255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c: Generated new TSK 0
I20260928 03:05:18.296203 15255 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
I20260928 03:05:18.371912 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41481
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:05:18.721732 15270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:18.722051 15270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:18.722149 15270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:18.730242 15270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:18.730382 15270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:18.730506 15270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:05:18.741576 15270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:18.742961 15270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:18.744585 15270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:18.759177 15275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:18.759196 15276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:18.759120 15278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:18.760761 15270 server_base.cc:1034] running on GCE node
I20260928 03:05:18.761561 15270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:18.762638 15270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:18.763870 15270 hybrid_clock.cc:648] HybridClock initialized: now 1790564718763803 us; error 63 us; skew 500 ppm
I20260928 03:05:18.764226 15270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:18 dist-test-slave-fjww krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564718, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:18.767733 15270 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:05:18.769480 15270 webserver.cc:460] Webserver started at http://127.6.192.1:38627/ using document root <none> and password file <none>
I20260928 03:05:18.770236 15270 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:18.770387 15270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:18.770711 15270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:18.773898 15270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "b27d4dc81ec0489ba132d5e435915cb2"
format_stamp: "Formatted at 2026-09-28 03:05:18 on dist-test-slave-fjww"
server_key: "64683e7c48f4aae9e5617f90bda42679"
server_key_iv: "5141fe92d63d14a8451e8329537808d4"
server_key_version: "encryptionkey@0"
I20260928 03:05:18.774745 15270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "b27d4dc81ec0489ba132d5e435915cb2"
format_stamp: "Formatted at 2026-09-28 03:05:18 on dist-test-slave-fjww"
server_key: "64683e7c48f4aae9e5617f90bda42679"
server_key_iv: "5141fe92d63d14a8451e8329537808d4"
server_key_version: "encryptionkey@0"
I20260928 03:05:18.780540 15270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 03:05:18.784561 15285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:18.786054 15270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:05:18.786257 15270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "b27d4dc81ec0489ba132d5e435915cb2"
format_stamp: "Formatted at 2026-09-28 03:05:18 on dist-test-slave-fjww"
server_key: "64683e7c48f4aae9e5617f90bda42679"
server_key_iv: "5141fe92d63d14a8451e8329537808d4"
server_key_version: "encryptionkey@0"
I20260928 03:05:18.786522 15270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:18.810436 15270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:18.814813 15270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:18.815117 15270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:18.816301 15270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:05:18.817730 15270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:18.817832 15270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:18.817975 15270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:18.818045 15270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:18.853528 15270 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:42937
I20260928 03:05:18.853578 15397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:42937 every 8 connection(s)
I20260928 03:05:18.854857 15270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:05:18.860095 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 15270
I20260928 03:05:18.860280 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:05:18.860899 6912 external_mini_cluster.cc:1428] Setting key 4e42145662de80c3cf4b55ba978e0c53
Sep 28 03:05:18 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564718, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:05:18.877138 15400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:18.857604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:38809 (local address 127.6.192.62:41481)
0928 03:05:18.858050 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:18.858054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:18.859140 (+ 1086us) server_negotiation.cc:407] Connection header received
0928 03:05:18.861578 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:18.861582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:18.861680 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:18.861786 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:18.863919 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:18.864618 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:18.866243 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:18.866666 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:18.866816 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:18.870797 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:18.870803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:18.870806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:18.870841 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:18.873576 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:18.874424 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:18.874429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:18.874431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:18.874560 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:18.875030 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:18.875039 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:18.875042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:18.875324 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:18.875519 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:18.876319 (+ 800us) server_negotiation.cc:299] Negotiation successful
0928 03:05:18.876733 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":216,"threads_started":1}
I20260928 03:05:18.878304 15401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:18.858206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.1:38809)
0928 03:05:18.858960 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:18.859005 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:18.861240 (+ 2235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:18.862105 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:18.862118 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:18.862954 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:18.863700 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:18.863721 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:18.864789 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:18.864801 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:18.866091 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:18.866105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:18.866910 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:18.866920 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:18.867179 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:18.867941 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:18.867969 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:18.870538 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:18.873752 (+ 3214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:18.873762 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:18.873782 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:18.874267 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:18.874690 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:18.874695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:18.874699 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:18.874906 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:18.875491 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:18.875501 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:18.875918 (+ 417us) client_negotiation.cc:769] Sending connection context
0928 03:05:18.876194 (+ 276us) client_negotiation.cc:240] Negotiation successful
0928 03:05:18.877002 (+ 808us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":300,"threads_started":1}
I20260928 03:05:18.880026 15398 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41481
I20260928 03:05:18.880360 15398 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:18.881191 15398 heartbeater.cc:511] Master 127.6.192.62:41481 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.192.2@KRBTEST.COM; defaulting to no policy
I20260928 03:05:18.883373 15209 ts_manager.cc:194] Registered new tserver with Master: b27d4dc81ec0489ba132d5e435915cb2 (127.6.192.1:42937)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.2@KRBTEST.COM" created.
I20260928 03:05:18.885035 15209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:38809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
I20260928 03:05:18.911783 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41481
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:05:19.250936 15404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:19.251261 15404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:19.251461 15404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:19.258893 15404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:19.259022 15404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:19.259204 15404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:05:19.270740 15404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:19.272203 15404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:19.273576 15404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:19.287559 15410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:19.287585 15409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:19.288193 15412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:19.288496 15404 server_base.cc:1034] running on GCE node
I20260928 03:05:19.289194 15404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:19.290412 15404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:19.291623 15404 hybrid_clock.cc:648] HybridClock initialized: now 1790564719291589 us; error 42 us; skew 500 ppm
I20260928 03:05:19.291997 15404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:19 dist-test-slave-fjww krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564719, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:19.295655 15404 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:05:19.297240 15404 webserver.cc:460] Webserver started at http://127.6.192.2:35869/ using document root <none> and password file <none>
I20260928 03:05:19.297987 15404 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:19.298118 15404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:19.298460 15404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:19.301653 15404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "68b41fe297e643ab98c4290b04d28dc8"
format_stamp: "Formatted at 2026-09-28 03:05:19 on dist-test-slave-fjww"
server_key: "83c9b278b14b6f3798abe7a2bd3410bf"
server_key_iv: "f423b39cf53b8c0dbbeff70ed3fcf049"
server_key_version: "encryptionkey@0"
I20260928 03:05:19.302488 15404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "68b41fe297e643ab98c4290b04d28dc8"
format_stamp: "Formatted at 2026-09-28 03:05:19 on dist-test-slave-fjww"
server_key: "83c9b278b14b6f3798abe7a2bd3410bf"
server_key_iv: "f423b39cf53b8c0dbbeff70ed3fcf049"
server_key_version: "encryptionkey@0"
I20260928 03:05:19.308204 15404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 03:05:19.312009 15419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:19.314152 15404 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 03:05:19.314356 15404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "68b41fe297e643ab98c4290b04d28dc8"
format_stamp: "Formatted at 2026-09-28 03:05:19 on dist-test-slave-fjww"
server_key: "83c9b278b14b6f3798abe7a2bd3410bf"
server_key_iv: "f423b39cf53b8c0dbbeff70ed3fcf049"
server_key_version: "encryptionkey@0"
I20260928 03:05:19.314641 15404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:19.367264 15404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:19.372890 15404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:19.373186 15404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:19.374253 15404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:05:19.375864 15404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:19.375946 15404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:19.376039 15404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:19.376086 15404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:19.413456 15404 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:33971
I20260928 03:05:19.413545 15531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:33971 every 8 connection(s)
I20260928 03:05:19.414892 15404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:05:19.415333 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 15404
I20260928 03:05:19.415529 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:05:19.416196 6912 external_mini_cluster.cc:1428] Setting key a9e398529b61451db281cd88971e3a95
Sep 28 03:05:19 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564719, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.192.3@KRBTEST.COM; defaulting to no policy
I20260928 03:05:19.438813 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:19.418071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:50465 (local address 127.6.192.62:41481)
0928 03:05:19.418557 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:19.418564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:19.419823 (+ 1259us) server_negotiation.cc:407] Connection header received
0928 03:05:19.421993 (+ 2170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:19.421999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:19.422107 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:19.422233 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:19.424263 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:19.425118 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:19.426883 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:19.427371 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:19.427602 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:19.431953 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:19.431959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:19.431963 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:19.432002 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:19.435059 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:19.436032 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:19.436037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:19.436040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:19.436213 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:19.436731 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:19.436737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:19.436740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:19.437050 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:19.437218 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:19.438105 (+ 887us) server_negotiation.cc:299] Negotiation successful
0928 03:05:19.438365 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":244,"threads_started":1}
I20260928 03:05:19.440059 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:19.418701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.2:50465)
0928 03:05:19.419643 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:19.419715 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:19.421678 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:19.422511 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:19.422521 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:19.423310 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:19.424059 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:19.424080 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:19.425298 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:19.425303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:19.426716 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:19.426730 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:19.427731 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:19.427741 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:19.428030 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:19.428799 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:19.428826 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:19.431651 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:19.435250 (+ 3599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:19.435262 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:19.435289 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:19.435864 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:19.436348 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:19.436358 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:19.436361 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:19.436594 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:19.437191 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:19.437200 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:19.437692 (+ 492us) client_negotiation.cc:769] Sending connection context
0928 03:05:19.437985 (+ 293us) client_negotiation.cc:240] Negotiation successful
0928 03:05:19.438621 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":342,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.3@KRBTEST.COM" created.
I20260928 03:05:19.441989 15532 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41481
I20260928 03:05:19.442478 15532 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:19.443485 15532 heartbeater.cc:511] Master 127.6.192.62:41481 requested a full tablet report, sending...
I20260928 03:05:19.445309 15209 ts_manager.cc:194] Registered new tserver with Master: 68b41fe297e643ab98c4290b04d28dc8 (127.6.192.2:33971)
I20260928 03:05:19.446266 15209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:50465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
I20260928 03:05:19.472299 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41481
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:05:19.805828 15538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:19.806124 15538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:19.806278 15538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:19.813894 15538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:19.814028 15538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:19.814200 15538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:05:19.824651 15538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:19.826074 15538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:19.827769 15538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:19.843357 15543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:19.843320 15544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:19.843393 15546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:19.844602 15538 server_base.cc:1034] running on GCE node
I20260928 03:05:19.845389 15538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:19.846524 15538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:19.847759 15538 hybrid_clock.cc:648] HybridClock initialized: now 1790564719847710 us; error 42 us; skew 500 ppm
I20260928 03:05:19.848121 15538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:19 dist-test-slave-fjww krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564719, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:19.851616 15538 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:05:19.853403 15538 webserver.cc:460] Webserver started at http://127.6.192.3:37887/ using document root <none> and password file <none>
I20260928 03:05:19.854346 15538 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:19.854501 15538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:19.854864 15538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:19.859088 15538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "092df80a79054a03bde32d55c153a704"
format_stamp: "Formatted at 2026-09-28 03:05:19 on dist-test-slave-fjww"
server_key: "eefc732cf0c030823e078343b455e8f1"
server_key_iv: "cdce1d3d70ebc7ebed2ea72e8b680884"
server_key_version: "encryptionkey@0"
I20260928 03:05:19.860100 15538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "092df80a79054a03bde32d55c153a704"
format_stamp: "Formatted at 2026-09-28 03:05:19 on dist-test-slave-fjww"
server_key: "eefc732cf0c030823e078343b455e8f1"
server_key_iv: "cdce1d3d70ebc7ebed2ea72e8b680884"
server_key_version: "encryptionkey@0"
I20260928 03:05:19.866143 15538 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260928 03:05:19.869890 15553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:19.871629 15538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 03:05:19.871769 15538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "092df80a79054a03bde32d55c153a704"
format_stamp: "Formatted at 2026-09-28 03:05:19 on dist-test-slave-fjww"
server_key: "eefc732cf0c030823e078343b455e8f1"
server_key_iv: "cdce1d3d70ebc7ebed2ea72e8b680884"
server_key_version: "encryptionkey@0"
I20260928 03:05:19.871963 15538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:19.888193 15398 heartbeater.cc:503] Master 127.6.192.62:41481 was elected leader, sending a full tablet report...
I20260928 03:05:19.901062 15538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:19.906039 15538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:19.906358 15538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:19.907395 15538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:05:19.908915 15538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:19.909021 15538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:19.909160 15538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:19.909229 15538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:19.946893 15538 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:42363
I20260928 03:05:19.946928 15665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:42363 every 8 connection(s)
I20260928 03:05:19.948648 15538 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:05:19.956200 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 15538
I20260928 03:05:19.956383 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:05:19.956820 6912 external_mini_cluster.cc:1428] Setting key c4d65906daea1aa8142da9699e7fc2db
Sep 28 03:05:19 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564719, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:05:19.971517 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:19.951612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:40679 (local address 127.6.192.62:41481)
0928 03:05:19.952115 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:19.952120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:19.953085 (+ 965us) server_negotiation.cc:407] Connection header received
0928 03:05:19.955266 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:19.955273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:19.955376 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:19.955544 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:19.957441 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:19.958617 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:19.960447 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:19.960875 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:19.961024 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:19.965287 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:19.965293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:19.965296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:19.965335 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:19.968171 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:19.969042 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:19.969047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:19.969049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:19.969170 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:19.969620 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:19.969630 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:19.969632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:19.969910 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:19.970078 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:19.970828 (+ 750us) server_negotiation.cc:299] Negotiation successful
0928 03:05:19.971079 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":259,"threads_started":1}
I20260928 03:05:19.972651 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:19.952115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.3:40679)
0928 03:05:19.952904 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:19.952954 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:19.954949 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:19.955830 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:19.955840 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:19.956548 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:19.957242 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:19.957262 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:19.958791 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:19.958796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:19.960281 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:19.960304 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:19.961133 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:19.961142 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:19.961399 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:19.962067 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:19.962092 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:19.965027 (+ 2935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:19.968339 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:19.968349 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:19.968370 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:19.968900 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:19.969286 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:19.969294 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:19.969297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:19.969496 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:19.970037 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:19.970047 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:19.970420 (+ 373us) client_negotiation.cc:769] Sending connection context
0928 03:05:19.970703 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 03:05:19.971348 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":310,"threads_started":1}
I20260928 03:05:19.974081 15666 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41481
I20260928 03:05:19.974406 15666 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:19.975179 15666 heartbeater.cc:511] Master 127.6.192.62:41481 requested a full tablet report, sending...
I20260928 03:05:19.976794 15209 ts_manager.cc:194] Registered new tserver with Master: 092df80a79054a03bde32d55c153a704 (127.6.192.3:42363)
I20260928 03:05:19.977638 15209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:40679
I20260928 03:05:19.984974 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.1.keytab.
Entry for principal oryx/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.1.keytab.
WARNING: no policy specified for oryx/127.6.192.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.192.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.2.keytab.
Entry for principal oryx/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.2.keytab.
WARNING: no policy specified for oryx/127.6.192.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.192.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.3.keytab.
Entry for principal oryx/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.3.keytab.
I20260928 03:05:20.136068 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 15270
Sep 28 03:05:20 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:05:20.155145 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:20.137615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.6.192.1:42937)
0928 03:05:20.138283 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:20.138291 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:20.138319 (+ 28us) server_negotiation.cc:407] Connection header received
0928 03:05:20.138400 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:20.138410 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:20.138661 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:20.138818 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:20.140413 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:20.141752 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:20.143222 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:20.144124 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:20.144257 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:20.148902 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:20.148930 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:20.148946 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:20.148995 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:20.152182 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:20.152914 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:20.152922 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:20.152925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:20.153063 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:20.153464 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:20.153468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:20.153470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:20.153856 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:20.154099 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:20.154485 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 03:05:20.154692 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":338,"threads_started":1}
I20260928 03:05:20.156597 15393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:20.249989 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 15270
I20260928 03:05:20.268208 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 15404
Sep 28 03:05:20 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.2@KRBTEST.COM
I20260928 03:05:20.286573 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:20.269231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57012 (local address 127.6.192.2:33971)
0928 03:05:20.269810 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:20.269818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:20.269843 (+ 25us) server_negotiation.cc:407] Connection header received
0928 03:05:20.269925 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:20.269931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:20.270144 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:20.270270 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:20.271676 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:20.272685 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:20.274452 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:20.275091 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:20.275239 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:20.280062 (+ 4823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:20.280078 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:20.280094 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:20.280147 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:20.283397 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:20.284288 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:20.284296 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:20.284298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:20.284451 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:20.284836 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:20.284840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:20.284842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:20.285207 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:20.285444 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:20.286012 (+ 568us) server_negotiation.cc:299] Negotiation successful
0928 03:05:20.286235 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":341,"threads_started":1}
I20260928 03:05:20.288151 15527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:20.373687 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 15404
I20260928 03:05:20.389917 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 15538
Sep 28 03:05:20 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.3@KRBTEST.COM
I20260928 03:05:20.408802 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:20.390840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.6.192.3:42363)
0928 03:05:20.391514 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:20.391522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:20.391572 (+ 50us) server_negotiation.cc:407] Connection header received
0928 03:05:20.391639 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:20.391646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:20.391889 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:20.392015 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:20.393356 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:20.394540 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:20.396121 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:20.396809 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:20.396956 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:20.401945 (+ 4989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:20.401960 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:20.401975 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:20.402023 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:20.405376 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:20.406279 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:20.406287 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:20.406289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:20.406500 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:20.406903 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:20.406908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:20.406911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:20.407301 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:20.407671 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:20.408119 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 03:05:20.408420 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":272,"threads_started":1}
I20260928 03:05:20.410189 15661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:20.499253 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 15538
I20260928 03:05:20.515084 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 15177
I20260928 03:05:20.516522 15239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:20.621959 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 15177
I20260928 03:05:20.644486 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:41481
--webserver_interface=127.6.192.62
--webserver_port=42825
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:41481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:05:20.998015 15684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:20.998418 15684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:20.998600 15684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:21.006999 15684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:05:21.007149 15684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:21.007254 15684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:21.007364 15684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:05:21.007470 15684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:05:21.018011 15684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:41481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:41481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=42825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:21.019383 15684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:21.020749 15684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:21.034893 15689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:21.034824 15690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:21.034824 15692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:21.045959 15684 server_base.cc:1034] running on GCE node
I20260928 03:05:21.047043 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:21.048925 15684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:21.050205 15684 hybrid_clock.cc:648] HybridClock initialized: now 1790564721050149 us; error 48 us; skew 500 ppm
I20260928 03:05:21.050650 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:21 dist-test-slave-fjww krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564721, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:21.054333 15684 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:05:21.056227 15684 webserver.cc:460] Webserver started at http://127.6.192.62:42825/ using document root <none> and password file <none>
I20260928 03:05:21.056993 15684 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:21.057152 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:21.063117 15684 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 03:05:21.066829 15699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:21.068674 15684 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:05:21.068893 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "0841e77eb42946fd975f3e72c0a51c0c"
format_stamp: "Formatted at 2026-09-28 03:05:18 on dist-test-slave-fjww"
server_key: "bf11e8f940c33fbd99e716603da9a6cf"
server_key_iv: "b632de68c96496c47d8688cbf987cd5e"
server_key_version: "encryptionkey@0"
I20260928 03:05:21.069973 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:21.089630 15684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:21.094786 15684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:21.095110 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:21.116954 15684 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:41481
I20260928 03:05:21.116988 15750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:41481 every 8 connection(s)
I20260928 03:05:21.118700 15684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:05:21.119275 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 15684
I20260928 03:05:21.120342 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:42937
--local_ip_for_outbound_sockets=127.6.192.1
--tserver_master_addrs=127.6.192.62:41481
--webserver_port=38627
--webserver_interface=127.6.192.1
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.192.1
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:05:21.126912 15751 sys_catalog.cc:263] Verifying existing consensus state
I20260928 03:05:21.133450 15751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c: Bootstrap starting.
I20260928 03:05:21.159971 15751 log.cc:824] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c: Log is configured to *not* fsync() on all Append() calls
I20260928 03:05:21.169900 15751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 03:05:21.170639 15751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c: Bootstrap complete.
I20260928 03:05:21.181568 15751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0841e77eb42946fd975f3e72c0a51c0c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41481 } }
I20260928 03:05:21.182595 15751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0841e77eb42946fd975f3e72c0a51c0c, State: Initialized, Role: FOLLOWER
I20260928 03:05:21.183266 15751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0841e77eb42946fd975f3e72c0a51c0c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41481 } }
I20260928 03:05:21.183563 15751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:05:21.183735 15751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:05:21.183944 15751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [term 1 FOLLOWER]: Advancing to term 2
I20260928 03:05:21.188320 15751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0841e77eb42946fd975f3e72c0a51c0c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41481 } }
I20260928 03:05:21.188881 15751 leader_election.cc:302] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0841e77eb42946fd975f3e72c0a51c0c; no voters:
I20260928 03:05:21.189615 15751 leader_election.cc:288] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 03:05:21.189761 15755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [term 2 FOLLOWER]: Leader election won for term 2
I20260928 03:05:21.191125 15755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [term 2 LEADER]: Becoming Leader. State: Replica: 0841e77eb42946fd975f3e72c0a51c0c, State: Running, Role: LEADER
I20260928 03:05:21.191854 15755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0841e77eb42946fd975f3e72c0a51c0c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41481 } }
I20260928 03:05:21.192207 15751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:05:21.194904 15756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0841e77eb42946fd975f3e72c0a51c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0841e77eb42946fd975f3e72c0a51c0c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41481 } } }
I20260928 03:05:21.195412 15756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [sys.catalog]: This master's current role is: LEADER
I20260928 03:05:21.195475 15757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0841e77eb42946fd975f3e72c0a51c0c. Latest consensus state: current_term: 2 leader_uuid: "0841e77eb42946fd975f3e72c0a51c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0841e77eb42946fd975f3e72c0a51c0c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41481 } } }
I20260928 03:05:21.195852 15757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c [sys.catalog]: This master's current role is: LEADER
I20260928 03:05:21.200508 15761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:05:21.207489 15761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:05:21.209120 15761 catalog_manager.cc:1259] Loaded cluster ID: 5194ce479e13435293817595c7c7cff1
I20260928 03:05:21.209316 15761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:05:21.213847 15761 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:05:21.215617 15761 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0841e77eb42946fd975f3e72c0a51c0c: Loaded TSK: 0
I20260928 03:05:21.217481 15761 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 03:05:21.496104 15753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:21.496464 15753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:21.496623 15753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:21.504570 15753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:21.504738 15753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:21.504910 15753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:05:21.515162 15753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.1.keytab
--principal=oryx/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:42937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=38627
--tserver_master_addrs=127.6.192.62:41481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:21.516618 15753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:21.518232 15753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:21.534010 15777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:21.534320 15778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:21.535202 15780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:21.535912 15753 server_base.cc:1034] running on GCE node
I20260928 03:05:21.536684 15753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:21.537757 15753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:21.539011 15753 hybrid_clock.cc:648] HybridClock initialized: now 1790564721538964 us; error 41 us; skew 500 ppm
I20260928 03:05:21.539357 15753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:21 dist-test-slave-fjww krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564721, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:21.542794 15753 init.cc:377] Logged in from keytab as oryx/127.6.192.1@KRBTEST.COM (short username oryx)
I20260928 03:05:21.544559 15753 webserver.cc:460] Webserver started at http://127.6.192.1:38627/ using document root <none> and password file <none>
I20260928 03:05:21.545295 15753 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:21.545459 15753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:21.551271 15753 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:05:21.554174 15787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:21.555763 15753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:05:21.555930 15753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "b27d4dc81ec0489ba132d5e435915cb2"
format_stamp: "Formatted at 2026-09-28 03:05:18 on dist-test-slave-fjww"
server_key: "64683e7c48f4aae9e5617f90bda42679"
server_key_iv: "5141fe92d63d14a8451e8329537808d4"
server_key_version: "encryptionkey@0"
I20260928 03:05:21.556928 15753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:21.573661 15753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:21.578531 15753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:21.578835 15753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:21.580024 15753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:05:21.581514 15753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:21.581611 15753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:21.581748 15753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:21.581847 15753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:21.618965 15753 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:42937
I20260928 03:05:21.619095 15899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:42937 every 8 connection(s)
I20260928 03:05:21.620721 15753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:05:21.629752 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 15753
I20260928 03:05:21.630548 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:33971
--local_ip_for_outbound_sockets=127.6.192.2
--tserver_master_addrs=127.6.192.62:41481
--webserver_port=35869
--webserver_interface=127.6.192.2
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.192.2
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:05:21 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:21 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 03:05:21.642961 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:21.624049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:38127 (local address 127.6.192.62:41481)
0928 03:05:21.624867 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:21.624885 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:21.625198 (+ 313us) server_negotiation.cc:407] Connection header received
0928 03:05:21.627421 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:21.627485 (+ 64us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:21.627947 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:21.628281 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:21.630494 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:21.631967 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:21.634707 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:21.635549 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:21.635706 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:21.642031 (+ 6325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.1:38127: BlockingRecv error: recv got EOF from 127.6.192.1:38127 (error 108)
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":291,"threads_started":1}
I20260928 03:05:21.642999 15903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:21.624154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.1:38127)
0928 03:05:21.625015 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:21.625060 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:21.626773 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:21.628534 (+ 1761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:21.628545 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:21.629245 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:21.630214 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:21.630243 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:21.632164 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:21.632172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:21.634482 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:21.634506 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:21.635850 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:21.635860 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:21.636212 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:21.637027 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:21.637063 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:21.641629 (+ 4566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":425,"threads_started":1}
W20260928 03:05:21.643831 15900 heartbeater.cc:644] Failed to heartbeat to 127.6.192.62:41481 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.192.62:41481: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Sep 28 03:05:21 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:21 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:21.653787 15903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:21.644511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.1:33881)
0928 03:05:21.644707 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:21.644728 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:21.644846 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:21.645250 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:21.645254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:21.645618 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:21.645764 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:21.645772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:21.647328 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:21.647333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:21.648306 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:21.648319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:21.649422 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:21.649426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:21.649563 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:21.650239 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:21.650257 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:21.653550 (+ 3293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 03:05:21.654408 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:21.644705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:33881 (local address 127.6.192.62:41481)
0928 03:05:21.644922 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:21.644930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:21.644953 (+ 23us) server_negotiation.cc:407] Connection header received
0928 03:05:21.645018 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:21.645025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:21.645108 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:21.645212 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:21.645916 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:21.647173 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:21.648465 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:21.649267 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:21.649434 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:21.653899 (+ 4465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.1:33881: BlockingRecv error: recv got EOF from 127.6.192.1:33881 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 03:05:21 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:21 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:21.664397 15903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:21.654588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.1:35831)
0928 03:05:21.654872 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:21.654892 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:21.655015 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:21.655676 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:21.655679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:21.655960 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:21.656108 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:21.656116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:21.657985 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:21.657991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:21.658968 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:21.658984 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:21.660097 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:21.660101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:21.660264 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:21.660960 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:21.660979 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:21.664172 (+ 3193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155}
W20260928 03:05:21.664731 15900 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 03:05:21.664980 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:21.654763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:35831 (local address 127.6.192.62:41481)
0928 03:05:21.654953 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:21.654961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:21.654990 (+ 29us) server_negotiation.cc:407] Connection header received
0928 03:05:21.655220 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:21.655229 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:21.655370 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:21.655508 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:21.656273 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:21.657785 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:21.659130 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:21.659940 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:21.660088 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:21.664486 (+ 4398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.1:35831: BlockingRecv error: recv got EOF from 127.6.192.1:35831 (error 108)
Metrics: {"server-negotiator.queue_time_us":60,"thread_start_us":579,"threads_started":1}
W20260928 03:05:22.007200 15904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:22.007716 15904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:22.007839 15904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:22.016196 15904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:22.016345 15904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:22.016463 15904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:05:22.026716 15904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.2.keytab
--principal=oryx/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:33971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=35869
--tserver_master_addrs=127.6.192.62:41481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:22.028223 15904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:22.029656 15904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:22.044536 15910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:22.044569 15911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:22.044633 15913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:22.045332 15904 server_base.cc:1034] running on GCE node
I20260928 03:05:22.045984 15904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:22.046994 15904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:22.048247 15904 hybrid_clock.cc:648] HybridClock initialized: now 1790564722048165 us; error 64 us; skew 500 ppm
I20260928 03:05:22.048673 15904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:22 dist-test-slave-fjww krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564722, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:22.052696 15904 init.cc:377] Logged in from keytab as oryx/127.6.192.2@KRBTEST.COM (short username oryx)
I20260928 03:05:22.054561 15904 webserver.cc:460] Webserver started at http://127.6.192.2:35869/ using document root <none> and password file <none>
I20260928 03:05:22.055336 15904 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:22.055537 15904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:22.062526 15904 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 03:05:22.065763 15920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:22.067729 15904 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:05:22.068033 15904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "68b41fe297e643ab98c4290b04d28dc8"
format_stamp: "Formatted at 2026-09-28 03:05:19 on dist-test-slave-fjww"
server_key: "83c9b278b14b6f3798abe7a2bd3410bf"
server_key_iv: "f423b39cf53b8c0dbbeff70ed3fcf049"
server_key_version: "encryptionkey@0"
I20260928 03:05:22.069212 15904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:22.083998 15904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:22.089205 15904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:22.089543 15904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:22.090742 15904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:05:22.092317 15904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:22.092418 15904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:22.092555 15904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:22.092636 15904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:22.131273 15904 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:33971
I20260928 03:05:22.131311 16032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:33971 every 8 connection(s)
I20260928 03:05:22.133378 15904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:05:22.139793 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 15904
I20260928 03:05:22.140467 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:42363
--local_ip_for_outbound_sockets=127.6.192.3
--tserver_master_addrs=127.6.192.62:41481
--webserver_port=37887
--webserver_interface=127.6.192.3
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.192.3
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:05:22 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:22 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 03:05:22.151688 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:22.136790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:42819 (local address 127.6.192.62:41481)
0928 03:05:22.137011 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:22.137016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:22.138348 (+ 1332us) server_negotiation.cc:407] Connection header received
0928 03:05:22.140545 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:22.140551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:22.140652 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:22.140751 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:22.142476 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.143217 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.144913 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.145335 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.145478 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:22.151320 (+ 5842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.2:42819: BlockingRecv error: recv got EOF from 127.6.192.2:42819 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 03:05:22.152330 16035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:22.137352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.2:42819)
0928 03:05:22.138172 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:22.138212 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:22.140225 (+ 2013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:22.141043 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:22.141054 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:22.141623 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:22.142263 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:22.142286 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.143401 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.143405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:22.144752 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:22.144764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.145595 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.145602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:22.145849 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:22.146614 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:22.146638 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:22.150985 (+ 4347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":340,"threads_started":1}
W20260928 03:05:22.153167 16033 heartbeater.cc:644] Failed to heartbeat to 127.6.192.62:41481 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.192.62:41481: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Sep 28 03:05:22 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:22 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:22.164007 16035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:22.153882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.2:42511)
0928 03:05:22.154097 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:22.154118 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:22.154235 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:22.154665 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:22.154669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:22.155020 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:22.155197 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:22.155206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.156913 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.156919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:22.158083 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:22.158098 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.159262 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.159267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:22.159542 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:22.160198 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:22.160217 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:22.163613 (+ 3396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 03:05:22.164024 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:22.153987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:42511 (local address 127.6.192.62:41481)
0928 03:05:22.154203 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:22.154211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:22.154237 (+ 26us) server_negotiation.cc:407] Connection header received
0928 03:05:22.154360 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:22.154374 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:22.154466 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:22.154587 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:22.155379 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.156698 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.158253 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.159056 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.159230 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:22.163840 (+ 4610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.2:42511: BlockingRecv error: recv got EOF from 127.6.192.2:42511 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 03:05:22 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:22 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:22.174086 16035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:22.164926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.2:56423)
0928 03:05:22.165098 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:22.165119 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:22.165229 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:22.165730 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:22.165735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:22.166146 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:22.166346 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:22.166359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.167768 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.167772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:22.168782 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:22.168795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.169706 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.169711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:22.169847 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:22.170534 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:22.170553 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:22.173717 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 03:05:22.174386 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:22.164997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:56423 (local address 127.6.192.62:41481)
0928 03:05:22.165157 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:22.165161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:22.165178 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:05:22.165511 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:22.165516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:22.165586 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:22.165681 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:22.166538 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.167547 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.168930 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.169560 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.169685 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:22.174019 (+ 4334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.2:56423: BlockingRecv error: recv got EOF from 127.6.192.2:56423 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 03:05:22.174625 16033 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 03:05:22.491767 16036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:22.492132 16036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:22.492329 16036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:22.499867 16036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:22.500038 16036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:22.500219 16036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:05:22.510555 16036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/krb5kdc/oryx_127.6.192.3.keytab
--principal=oryx/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:42363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=37887
--tserver_master_addrs=127.6.192.62:41481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:22.512085 16036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:22.513474 16036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:22.527884 16041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:22.527971 16042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:22.527923 16044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:22.529382 16036 server_base.cc:1034] running on GCE node
I20260928 03:05:22.530089 16036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:22.531339 16036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:22.532608 16036 hybrid_clock.cc:648] HybridClock initialized: now 1790564722532531 us; error 70 us; skew 500 ppm
I20260928 03:05:22.532999 16036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:22 dist-test-slave-fjww krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564722, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:22.537096 16036 init.cc:377] Logged in from keytab as oryx/127.6.192.3@KRBTEST.COM (short username oryx)
I20260928 03:05:22.538918 16036 webserver.cc:460] Webserver started at http://127.6.192.3:37887/ using document root <none> and password file <none>
I20260928 03:05:22.539777 16036 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:22.539928 16036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:22.545943 16036 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 03:05:22.549173 16051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:22.550989 16036 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:05:22.551231 16036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "092df80a79054a03bde32d55c153a704"
format_stamp: "Formatted at 2026-09-28 03:05:19 on dist-test-slave-fjww"
server_key: "eefc732cf0c030823e078343b455e8f1"
server_key_iv: "cdce1d3d70ebc7ebed2ea72e8b680884"
server_key_version: "encryptionkey@0"
I20260928 03:05:22.552484 16036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:22.574348 16036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:22.579276 16036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:22.579656 16036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:22.580741 16036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:05:22.582405 16036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:22.582506 16036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:22.582597 16036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:22.582659 16036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:22.619984 16036 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:42363
I20260928 03:05:22.620023 16163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:42363 every 8 connection(s)
I20260928 03:05:22.622097 16036 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:05:22.628487 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 16036
Sep 28 03:05:22 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:22 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 03:05:22.642606 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:22.625079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:59993 (local address 127.6.192.62:41481)
0928 03:05:22.625265 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:22.625269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:22.626860 (+ 1591us) server_negotiation.cc:407] Connection header received
0928 03:05:22.629163 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:22.629171 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:22.629282 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:22.629526 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:22.631657 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.632880 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.634572 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.635255 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.635403 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:22.642272 (+ 6869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.3:59993: BlockingRecv error: recv got EOF from 127.6.192.3:59993 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 03:05:22.643328 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:22.625804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.3:59993)
0928 03:05:22.626658 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:22.626706 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:22.628739 (+ 2033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:22.629819 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:22.629836 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:22.630800 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:22.631432 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:22.631509 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.633043 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.633047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:22.634426 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:22.634441 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.635598 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.635608 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:22.635877 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:22.636686 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:22.636726 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:22.641976 (+ 5250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":311,"threads_started":1}
W20260928 03:05:22.644181 16164 heartbeater.cc:644] Failed to heartbeat to 127.6.192.62:41481 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.192.62:41481: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
I20260928 03:05:22.647140 16168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:22.629700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38200 (local address 127.6.192.62:41481)
0928 03:05:22.630922 (+ 1222us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:22.630929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:22.630947 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:05:22.630998 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:22.631003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:22.631084 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:22.631209 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:22.633328 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.634113 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.635917 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.636465 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.636592 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:22.639807 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:22.639817 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:22.639837 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:22.639887 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:22.643309 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:22.644226 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:22.644233 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:22.644240 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:22.644467 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:22.644890 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:22.644896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:22.644899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:22.645467 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:22.645879 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:22.646367 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 03:05:22.646786 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1078,"thread_start_us":918,"threads_started":1}
Sep 28 03:05:22 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:22 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:22.653908 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:22.644832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.3:35453)
0928 03:05:22.645041 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:22.645061 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:22.645170 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:22.645510 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:22.645515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:22.645873 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:22.646020 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:22.646028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.647164 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.647171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:22.648217 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:22.648229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.649259 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.649264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:22.649522 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:22.650305 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:22.650324 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:22.653701 (+ 3377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 03:05:22.654218 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:22.644915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:35453 (local address 127.6.192.62:41481)
0928 03:05:22.645083 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:22.645091 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:22.645110 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:05:22.645299 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:22.645304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:22.645379 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:22.645481 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:22.646157 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.647029 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.648432 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.649082 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.649206 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:22.653966 (+ 4760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.3:35453: BlockingRecv error: recv got EOF from 127.6.192.3:35453 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 03:05:22 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:22 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:22.664141 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:22.654825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.3:43617)
0928 03:05:22.655009 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:22.655033 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:22.655131 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:22.655669 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:22.655674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:22.656080 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:22.656269 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:22.656279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.657698 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.657703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:22.658691 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:22.658704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.659382 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.659387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:22.659604 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:22.660236 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:22.660256 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:22.663763 (+ 3507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 03:05:22.664239 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:22.654921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:43617 (local address 127.6.192.62:41481)
0928 03:05:22.655165 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:22.655169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:22.655189 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:05:22.655377 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:22.655381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:22.655509 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:22.655600 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:22.656577 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.657442 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.658821 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.659232 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.659332 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:22.664009 (+ 4677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.3:43617: BlockingRecv error: recv got EOF from 127.6.192.3:43617 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
W20260928 03:05:22.664575 16164 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 03:05:22 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:22 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:22.674156 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:22.665320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.1:35661)
0928 03:05:22.665904 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:22.665920 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:22.666073 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:22.666392 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:22.666397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:22.666871 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:22.667070 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:22.667079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.668172 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.668178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:22.669295 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:22.669307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.670051 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.670057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:22.670227 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:22.670837 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:22.670855 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:22.673891 (+ 3036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":335,"threads_started":1}
W20260928 03:05:22.674455 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:22.665534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:35661 (local address 127.6.192.62:41481)
0928 03:05:22.665777 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:22.665783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:22.666000 (+ 217us) server_negotiation.cc:407] Connection header received
0928 03:05:22.666202 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:22.666206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:22.666265 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:22.666361 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:22.667211 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.668025 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.669440 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:22.669850 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:22.670000 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:22.674207 (+ 4207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.1:35661: BlockingRecv error: recv got EOF from 127.6.192.1:35661 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 28 03:05:23 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:23 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:23.191071 16170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:23.175328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.2:40903)
0928 03:05:23.175939 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:23.175957 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:23.176088 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:23.176481 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:23.176484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:23.176973 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:23.177182 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:23.177190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:23.178255 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:23.178260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:23.179709 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:23.179731 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:23.180503 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:23.180508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:23.180653 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:23.187649 (+ 6996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:23.187673 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:23.190803 (+ 3130us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":288,"threads_started":1}
W20260928 03:05:23.191411 16171 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:23.175406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:40903 (local address 127.6.192.62:41481)
0928 03:05:23.175950 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:23.175955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:23.176015 (+ 60us) server_negotiation.cc:407] Connection header received
0928 03:05:23.176174 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:23.176177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:23.176237 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:23.176328 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:23.177330 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:23.178078 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:23.179882 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:23.180301 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:23.180418 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:23.191079 (+ 10661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.2:40903: BlockingRecv error: recv got EOF from 127.6.192.2:40903 (error 108)
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":254,"threads_started":1}
Sep 28 03:05:23 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:23 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:23.674777 16172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:23.665320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.3:52239)
0928 03:05:23.665941 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:23.665961 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:23.666131 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:23.666515 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:23.666519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:23.667008 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:23.667193 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:23.667202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:23.668297 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:23.668303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:23.669258 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:23.669269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:23.670041 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:23.670047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:23.670224 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:23.670978 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:23.670996 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:23.674436 (+ 3440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":293,"threads_started":1}
W20260928 03:05:23.675027 16171 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:23.665414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:52239 (local address 127.6.192.62:41481)
0928 03:05:23.665660 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:23.665668 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:23.666059 (+ 391us) server_negotiation.cc:407] Connection header received
0928 03:05:23.666275 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:23.666280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:23.666358 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:23.666483 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:23.667345 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:23.668145 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:23.669412 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:23.669856 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:23.669996 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:23.674712 (+ 4716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.3:52239: BlockingRecv error: recv got EOF from 127.6.192.3:52239 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 03:05:23 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:23 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:23.683252 16173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:23.675201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.1:45559)
0928 03:05:23.675725 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:23.675740 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:23.675887 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:23.676231 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:23.676236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:23.676620 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:23.676768 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:23.676776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:23.677772 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:23.677778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:23.678699 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:23.678711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:23.679411 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:23.679417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:23.679610 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:23.680219 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:23.680236 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:23.682988 (+ 2752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":295,"threads_started":1}
W20260928 03:05:23.683584 16171 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:23.675334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:45559 (local address 127.6.192.62:41481)
0928 03:05:23.675574 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:23.675581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:23.675818 (+ 237us) server_negotiation.cc:407] Connection header received
0928 03:05:23.676010 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:23.676015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:23.676089 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:23.676186 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:23.676906 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:23.677632 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:23.678853 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:23.679257 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:23.679390 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:23.683268 (+ 3878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.1:45559: BlockingRecv error: recv got EOF from 127.6.192.1:45559 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 03:05:24 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:24 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:24.201591 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:24.192254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.2:39899)
0928 03:05:24.192985 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:24.193014 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:24.193171 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:24.193661 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:24.193668 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:24.194136 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:24.194292 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:24.194303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:24.195502 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:24.195507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:24.196508 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:24.196521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:24.197305 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:24.197312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:24.197505 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:24.198161 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:24.198180 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:24.201343 (+ 3163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":417,"threads_started":1}
W20260928 03:05:24.202039 16175 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:24.192429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:39899 (local address 127.6.192.62:41481)
0928 03:05:24.193308 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:24.193314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:24.193329 (+ 15us) server_negotiation.cc:407] Connection header received
0928 03:05:24.193383 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:24.193387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:24.193472 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:24.193603 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:24.194481 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:24.195309 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:24.196675 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:24.197132 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:24.197285 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:24.201595 (+ 4310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.2:39899: BlockingRecv error: recv got EOF from 127.6.192.2:39899 (error 108)
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":622,"threads_started":1}
Sep 28 03:05:24 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:24 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:24.686264 16176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:24.676470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.3:36865)
0928 03:05:24.677089 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:24.677114 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:24.677262 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:24.677684 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:24.677689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:24.678151 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:24.678312 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:24.678321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:24.679531 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:24.679539 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:24.680544 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:24.680556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:24.681359 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:24.681366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:24.681552 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:24.682248 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:24.682269 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:24.685951 (+ 3682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":378,"threads_started":1}
W20260928 03:05:24.686466 16175 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:24.676788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:36865 (local address 127.6.192.62:41481)
0928 03:05:24.677206 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:24.677213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:24.677236 (+ 23us) server_negotiation.cc:407] Connection header received
0928 03:05:24.677443 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:24.677456 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:24.677526 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:24.677647 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:24.678491 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:24.679332 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:24.680705 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:24.681180 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:24.681337 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:24.686197 (+ 4860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.3:36865: BlockingRecv error: recv got EOF from 127.6.192.3:36865 (error 108)
Metrics: {"server-negotiator.queue_time_us":270}
Sep 28 03:05:24 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:24 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:24.695225 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:24.684618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.1:37385)
0928 03:05:24.685272 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:24.685299 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:24.685454 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:24.685930 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:24.685934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:24.686490 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:24.686747 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:24.686758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:24.687988 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:24.687993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:24.689176 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:24.689191 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:24.690398 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:24.690404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:24.690620 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:24.691479 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:24.691507 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:24.694819 (+ 3312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":303,"threads_started":1}
W20260928 03:05:24.695489 16178 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:24.684707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:37385 (local address 127.6.192.62:41481)
0928 03:05:24.685564 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:24.685570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:24.685588 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:05:24.685646 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:24.685650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:24.685734 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:24.685862 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:24.686886 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:24.687826 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:24.689416 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:24.690172 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:24.690333 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:24.695118 (+ 4785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.1:37385: BlockingRecv error: recv got EOF from 127.6.192.1:37385 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":728,"thread_start_us":592,"threads_started":1}
Sep 28 03:05:25 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:25 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:25.212623 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:25.202687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.2:49037)
0928 03:05:25.203349 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:25.203368 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:25.203561 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:25.203973 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:25.203977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:25.204353 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:25.204538 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:25.204547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:25.205745 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:25.205750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:25.207203 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:25.207217 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:25.208065 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:25.208070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:25.208203 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:25.208901 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:25.208921 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:25.212316 (+ 3395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":361,"threads_started":1}
W20260928 03:05:25.212949 16180 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:25.202759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:49037 (local address 127.6.192.62:41481)
0928 03:05:25.203435 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:25.203484 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:25.203513 (+ 29us) server_negotiation.cc:407] Connection header received
0928 03:05:25.203680 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:25.203684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:25.203747 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:25.203863 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:25.204713 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:25.205601 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:25.207343 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:25.207895 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:25.208001 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:25.212595 (+ 4594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.2:49037: BlockingRecv error: recv got EOF from 127.6.192.2:49037 (error 108)
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":448,"threads_started":1}
Sep 28 03:05:25 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:25 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:25.697680 16181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:25.687486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.3:35989)
0928 03:05:25.688106 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:25.688126 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:25.688292 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:25.688731 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:25.688736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:25.689117 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:25.689295 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:25.689305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:25.690762 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:25.690768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:25.692135 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:25.692149 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:25.693062 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:25.693073 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:25.693233 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:25.693888 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:25.693906 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:25.697278 (+ 3372us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":277,"threads_started":1}
W20260928 03:05:25.697855 16180 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:25.687692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:35989 (local address 127.6.192.62:41481)
0928 03:05:25.687930 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:25.687936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:25.688228 (+ 292us) server_negotiation.cc:407] Connection header received
0928 03:05:25.688504 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:25.688508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:25.688575 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:25.688690 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:25.689486 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:25.690587 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:25.692384 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:25.692831 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:25.693024 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:25.697581 (+ 4557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.3:35989: BlockingRecv error: recv got EOF from 127.6.192.3:35989 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 03:05:25 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:25 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:25.707234 16182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:25.696316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.1:55955)
0928 03:05:25.697293 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:25.697314 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:25.697493 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:25.698117 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:25.698122 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:25.698619 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:25.698784 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:25.698793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:25.699960 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:25.699964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:25.701101 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:25.701115 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:25.702132 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:25.702137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:25.702288 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:25.703098 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:25.703123 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:25.706980 (+ 3857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":725,"threads_started":1}
W20260928 03:05:25.707657 16183 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:25.696427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:55955 (local address 127.6.192.62:41481)
0928 03:05:25.697084 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:25.697090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:25.697493 (+ 403us) server_negotiation.cc:407] Connection header received
0928 03:05:25.697648 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:25.697657 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:25.697727 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:25.697880 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:25.698934 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:25.699823 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:25.701481 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:25.701950 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:25.702053 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:25.707245 (+ 5192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.1:55955: BlockingRecv error: recv got EOF from 127.6.192.1:55955 (error 108)
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":272,"threads_started":1}
Sep 28 03:05:26 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:26 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:26.223345 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:26.213734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.2:51315)
0928 03:05:26.214599 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:26.214621 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:26.214754 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:26.215162 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:26.215167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:26.215744 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:26.215898 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:26.215907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:26.216979 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:26.216984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:26.218103 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:26.218117 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:26.218990 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:26.218995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:26.219143 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:26.219836 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:26.219862 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:26.223009 (+ 3147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":365,"threads_started":1}
W20260928 03:05:26.223753 16185 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:26.213857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:51315 (local address 127.6.192.62:41481)
0928 03:05:26.214571 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:26.214598 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:26.214704 (+ 106us) server_negotiation.cc:407] Connection header received
0928 03:05:26.214884 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:26.214890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:26.214976 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:26.215075 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:26.216042 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:26.216827 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:26.218332 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:26.218840 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:26.218959 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:26.223256 (+ 4297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.2:51315: BlockingRecv error: recv got EOF from 127.6.192.2:51315 (error 108)
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":548,"threads_started":1}
Sep 28 03:05:26 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:26 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:26.709002 16186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:26.699288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.3:51417)
0928 03:05:26.699888 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:26.699904 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:26.700072 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:26.700473 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:26.700477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:26.700982 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:26.701254 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:26.701273 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:26.702470 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:26.702475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:26.703571 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:26.703583 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:26.704280 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:26.704284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:26.704512 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:26.705114 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:26.705132 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:26.708658 (+ 3526us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":355,"threads_started":1}
W20260928 03:05:26.709296 16185 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:26.699558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:51417 (local address 127.6.192.62:41481)
0928 03:05:26.699791 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:26.699799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:26.699993 (+ 194us) server_negotiation.cc:407] Connection header received
0928 03:05:26.700215 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:26.700220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:26.700291 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:26.700402 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:26.701424 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:26.702325 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:26.703706 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:26.704137 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:26.704234 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:26.709016 (+ 4782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.3:51417: BlockingRecv error: recv got EOF from 127.6.192.3:51417 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 03:05:26 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:26 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:26.717751 16187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:26.708302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.1:53855)
0928 03:05:26.708990 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:26.709020 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:26.709191 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:26.709845 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:26.709849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:26.710261 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:26.710424 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:26.710434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:26.711521 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:26.711534 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:26.712745 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:26.712764 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:26.713722 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:26.713728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:26.713904 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:26.714582 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:26.714601 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:26.717400 (+ 2799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":250,"threads_started":1}
W20260928 03:05:26.717890 16185 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:26.708408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:53855 (local address 127.6.192.62:41481)
0928 03:05:26.709532 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:26.709538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:26.709558 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:05:26.709620 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:26.709627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:26.709720 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:26.709799 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:26.710563 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:26.711312 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:26.712915 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:26.713564 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:26.713695 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:26.717613 (+ 3918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.1:53855: BlockingRecv error: recv got EOF from 127.6.192.1:53855 (error 108)
Metrics: {"server-negotiator.queue_time_us":929,"thread_start_us":954,"threads_started":1}
Sep 28 03:05:27 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:27 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:27.234392 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:27.224345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.2:47279)
0928 03:05:27.224878 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:27.224895 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:27.224998 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:27.225598 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:27.225602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:27.225997 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:27.226171 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:27.226180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:27.227355 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:27.227360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:27.228643 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:27.228657 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:27.229495 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:27.229500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:27.229645 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:27.230346 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:27.230449 (+ 103us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:27.234040 (+ 3591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":263,"threads_started":1}
W20260928 03:05:27.234838 16190 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:27.224598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:47279 (local address 127.6.192.62:41481)
0928 03:05:27.225204 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:27.225209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:27.225226 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:05:27.225278 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:27.225282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:27.225343 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:27.225539 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:27.226306 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:27.227208 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:27.228793 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:27.229244 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:27.229443 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:27.234312 (+ 4869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.2:47279: BlockingRecv error: recv got EOF from 127.6.192.2:47279 (error 108)
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":389,"threads_started":1}
Sep 28 03:05:27 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:27 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:27.720204 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:27.710397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.3:43545)
0928 03:05:27.711068 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:27.711084 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:27.711199 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:27.711584 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:27.711589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:27.712032 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:27.712202 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:27.712211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:27.713448 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:27.713452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:27.714411 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:27.714424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:27.715251 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:27.715255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:27.715382 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:27.716187 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:27.716206 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:27.719835 (+ 3629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":283,"threads_started":1}
W20260928 03:05:27.720819 16190 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:27.710571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:43545 (local address 127.6.192.62:41481)
0928 03:05:27.710783 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:27.710788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:27.711166 (+ 378us) server_negotiation.cc:407] Connection header received
0928 03:05:27.711314 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:27.711319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:27.711384 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:27.711511 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:27.712369 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:27.713251 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:27.714639 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:27.715088 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:27.715202 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:27.720603 (+ 5401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.3:43545: BlockingRecv error: recv got EOF from 127.6.192.3:43545 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 28 03:05:27 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:27 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:27.728300 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:27.718798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.1:60277)
0928 03:05:27.719405 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:27.719428 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:27.719598 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:27.720204 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:27.720210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:27.720959 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:27.721181 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:27.721196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:27.722577 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:27.722582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:27.723582 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:27.723596 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:27.724395 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:27.724399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:27.724561 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:27.725226 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:27.725243 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:27.728075 (+ 2832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":345,"threads_started":1}
W20260928 03:05:27.728809 16193 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:27.718941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:60277 (local address 127.6.192.62:41481)
0928 03:05:27.719849 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:27.719855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:27.719872 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:05:27.719927 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:27.719931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:27.720017 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:27.720122 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:27.721361 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:27.722431 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:27.723758 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:27.724226 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:27.724385 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:27.728507 (+ 4122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.1:60277: BlockingRecv error: recv got EOF from 127.6.192.1:60277 (error 108)
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":608,"threads_started":1}
Sep 28 03:05:28 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:28 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:28.249145 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:28.235442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.2:34999)
0928 03:05:28.236068 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:28.236084 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:28.236200 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:28.236572 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:28.236576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:28.237012 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:28.237220 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:28.237231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:28.238445 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:28.238450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:28.239526 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:28.239542 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:28.240292 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:28.240298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:28.240522 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:28.245577 (+ 5055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:28.245607 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:28.248923 (+ 3316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":368,"threads_started":1}
W20260928 03:05:28.249455 16195 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:28.235681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:34999 (local address 127.6.192.62:41481)
0928 03:05:28.236148 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:28.236152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:28.236170 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:05:28.236329 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:28.236333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:28.236396 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:28.236487 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:28.237458 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:28.238201 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:28.239674 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:28.240090 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:28.240190 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:28.249142 (+ 8952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.2:34999: BlockingRecv error: recv got EOF from 127.6.192.2:34999 (error 108)
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":268,"threads_started":1}
Sep 28 03:05:28 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:28 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:28.731148 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:28.721288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.3:42747)
0928 03:05:28.721999 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:28.722017 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:28.722137 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:28.722514 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:28.722518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:28.722951 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:28.723122 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:28.723131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:28.724405 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:28.724409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:28.725415 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:28.725427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:28.726174 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:28.726179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:28.726313 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:28.726997 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:28.727016 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:28.730785 (+ 3769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":446,"threads_started":1}
W20260928 03:05:28.731245 16195 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:28.721395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:42747 (local address 127.6.192.62:41481)
0928 03:05:28.721570 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:28.721574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:28.722113 (+ 539us) server_negotiation.cc:407] Connection header received
0928 03:05:28.722245 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:28.722249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:28.722314 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:28.722481 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:28.723464 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:28.724231 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:28.725580 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:28.726008 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:28.726114 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:28.731012 (+ 4898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.3:42747: BlockingRecv error: recv got EOF from 127.6.192.3:42747 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 03:05:28 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:28 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:28.739257 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:28.729408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.1:54345)
0928 03:05:28.730053 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:28.730073 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:28.730195 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:28.730687 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:28.730706 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:28.731152 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:28.731413 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:28.731425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:28.732774 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:28.732779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:28.733741 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:28.733756 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:28.734729 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:28.734743 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:28.734876 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:28.735590 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:28.735615 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:28.738997 (+ 3382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":258,"threads_started":1}
W20260928 03:05:28.739665 16198 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:28.729624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:54345 (local address 127.6.192.62:41481)
0928 03:05:28.730384 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:28.730390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:28.730409 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:05:28.730457 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:28.730466 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:28.730525 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:28.730661 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:28.731578 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:28.732413 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:28.733910 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:28.734400 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:28.734503 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:28.739297 (+ 4794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.1:54345: BlockingRecv error: recv got EOF from 127.6.192.1:54345 (error 108)
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":467,"threads_started":1}
Sep 28 03:05:29 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:29 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:29.259788 16199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:29.250158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.2:50079)
0928 03:05:29.251094 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:29.251119 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:29.251236 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:29.251615 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:29.251618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:29.252047 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:29.252214 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:29.252223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:29.253329 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:29.253333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:29.254348 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:29.254361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:29.255108 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:29.255112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:29.255229 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:29.255929 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:29.255947 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:29.259553 (+ 3606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":599,"threads_started":1}
W20260928 03:05:29.260125 16200 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:29.250266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:50079 (local address 127.6.192.62:41481)
0928 03:05:29.250802 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:29.250807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:29.251207 (+ 400us) server_negotiation.cc:407] Connection header received
0928 03:05:29.251338 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:29.251342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:29.251415 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:29.251589 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:29.252365 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:29.253188 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:29.254513 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:29.254963 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:29.255067 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:29.259764 (+ 4697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.2:50079: BlockingRecv error: recv got EOF from 127.6.192.2:50079 (error 108)
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":324,"threads_started":1}
Sep 28 03:05:29 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:29 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:29.742708 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:29.732278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.3:42613)
0928 03:05:29.733025 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:29.733041 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:29.733166 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:29.733681 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:29.733685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:29.734120 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:29.734290 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:29.734299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:29.735575 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:29.735579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:29.736578 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:29.736591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:29.737323 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:29.737328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:29.737555 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:29.738335 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:29.738360 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:29.742327 (+ 3967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":281,"threads_started":1}
W20260928 03:05:29.742725 16200 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:29.732401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:42613 (local address 127.6.192.62:41481)
0928 03:05:29.732643 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:29.732647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:29.733142 (+ 495us) server_negotiation.cc:407] Connection header received
0928 03:05:29.733282 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:29.733286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:29.733353 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:29.733447 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:29.734475 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:29.735226 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:29.736725 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:29.737150 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:29.737250 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:29.742535 (+ 5285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.3:42613: BlockingRecv error: recv got EOF from 127.6.192.3:42613 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 28 03:05:29 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:29 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:29.750567 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:29.740332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.1:53565)
0928 03:05:29.741561 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:29.741582 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:29.741710 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:29.742238 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:29.742246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:29.742746 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:29.742960 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:29.742974 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:29.744139 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:29.744144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:29.745207 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:29.745219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:29.746073 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:29.746081 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:29.746274 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:29.746887 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:29.746904 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:29.750312 (+ 3408us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1059,"thread_start_us":933,"threads_started":1}
W20260928 03:05:29.750993 16203 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:29.740506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:53565 (local address 127.6.192.62:41481)
0928 03:05:29.741953 (+ 1447us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:29.741957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:29.741973 (+ 16us) server_negotiation.cc:407] Connection header received
0928 03:05:29.742016 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:29.742019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:29.742075 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:29.742161 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:29.743141 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:29.743979 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:29.745394 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:29.745855 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:29.745992 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:29.750687 (+ 4695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.1:53565: BlockingRecv error: recv got EOF from 127.6.192.1:53565 (error 108)
Metrics: {"server-negotiator.queue_time_us":1335,"thread_start_us":1206,"threads_started":1}
Sep 28 03:05:30 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:30 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:30.270975 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:30.261023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.2:51165)
0928 03:05:30.261671 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:30.261690 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:30.261813 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:30.262237 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:30.262241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:30.262706 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:30.262872 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:30.262882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:30.264355 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:30.264364 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:30.265613 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:30.265632 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:30.266653 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:30.266659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:30.266796 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:30.267581 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:30.267602 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:30.270710 (+ 3108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":349,"threads_started":1}
W20260928 03:05:30.271589 16205 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:30.261174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:51165 (local address 127.6.192.62:41481)
0928 03:05:30.261760 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:30.261765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:30.261786 (+ 21us) server_negotiation.cc:407] Connection header received
0928 03:05:30.261997 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:30.262002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:30.262067 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:30.262191 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:30.263045 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:30.264191 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:30.265804 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:30.266457 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:30.266576 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:30.271046 (+ 4470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.2:51165: BlockingRecv error: recv got EOF from 127.6.192.2:51165 (error 108)
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":291,"threads_started":1}
Sep 28 03:05:30 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:30 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:30.753451 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:30.743776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.3:56319)
0928 03:05:30.744340 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:30.744357 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:30.744490 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:30.744831 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:30.744835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:30.745224 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:30.745374 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:30.745382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:30.746566 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:30.746570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:30.747605 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:30.747619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:30.748420 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:30.748425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:30.748584 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:30.749220 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:30.749240 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:30.753124 (+ 3884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":304,"threads_started":1}
W20260928 03:05:30.753530 16205 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:30.743918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:56319 (local address 127.6.192.62:41481)
0928 03:05:30.744091 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:30.744095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:30.744475 (+ 380us) server_negotiation.cc:407] Connection header received
0928 03:05:30.744620 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:30.744625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:30.744695 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:30.744787 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:30.745567 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:30.746300 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:30.747751 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:30.748201 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:30.748315 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:30.753341 (+ 5026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.3:56319: BlockingRecv error: recv got EOF from 127.6.192.3:56319 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 03:05:30 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:30 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:30.762568 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:30.751592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.1:51897)
0928 03:05:30.752258 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:30.752284 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:30.752630 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:30.753746 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:30.753753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:30.754191 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:30.754393 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:30.754402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:30.755694 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:30.755698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:30.756773 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:30.756785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:30.757661 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:30.757666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:30.757798 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:30.758638 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:30.758657 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:30.762150 (+ 3493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":372,"threads_started":1}
W20260928 03:05:30.762861 16208 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:30.751785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:51897 (local address 127.6.192.62:41481)
0928 03:05:30.752582 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:30.752588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:30.752621 (+ 33us) server_negotiation.cc:407] Connection header received
0928 03:05:30.752791 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:30.752797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:30.752879 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:30.753005 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:30.754556 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:30.755560 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:30.756916 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:30.757510 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:30.757611 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:30.762539 (+ 4928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.1:51897: BlockingRecv error: recv got EOF from 127.6.192.1:51897 (error 108)
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":496,"threads_started":1}
Sep 28 03:05:31 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:31 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:31.282996 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:31.272237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.2:58933)
0928 03:05:31.272817 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:31.272835 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:31.272972 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:31.273514 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:31.273518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:31.274041 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:31.274216 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:31.274226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:31.276046 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:31.276052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:31.277396 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:31.277410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:31.278208 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:31.278214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:31.278355 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:31.279121 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:31.279142 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:31.282717 (+ 3575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":330,"threads_started":1}
W20260928 03:05:31.283351 16210 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:31.272357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:58933 (local address 127.6.192.62:41481)
0928 03:05:31.273184 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:31.273188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:31.273206 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:05:31.273259 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:31.273262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:31.273318 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:31.273413 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:31.274551 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:31.275787 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:31.277563 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:31.277990 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:31.278117 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:31.282996 (+ 4879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.2:58933: BlockingRecv error: recv got EOF from 127.6.192.2:58933 (error 108)
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":598,"threads_started":1}
Sep 28 03:05:31 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:31 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 03:05:31.764844 16210 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:31.754669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:40215 (local address 127.6.192.62:41481)
0928 03:05:31.754912 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:31.754922 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:31.755237 (+ 315us) server_negotiation.cc:407] Connection header received
0928 03:05:31.755464 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:31.755469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:31.755539 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:31.755628 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:31.756619 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:31.757501 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:31.759116 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:31.759664 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:31.759824 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:31.764623 (+ 4799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.3:40215: BlockingRecv error: recv got EOF from 127.6.192.3:40215 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 03:05:31.765508 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:31.754636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.3:40215)
0928 03:05:31.755117 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:31.755133 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:31.755259 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:31.755687 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:31.755691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:31.756184 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:31.756466 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:31.756478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:31.757653 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:31.757658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:31.758952 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:31.758968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:31.759850 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:31.759855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:31.760073 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:31.760829 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:31.760852 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:31.765127 (+ 4275us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":263,"threads_started":1}
Sep 28 03:05:31 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:31 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:31.774151 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:31.763724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.1:42039)
0928 03:05:31.764270 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:31.764290 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:31.764413 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:31.765244 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:31.765249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:31.765772 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:31.765945 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:31.765954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:31.767172 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:31.767176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:31.768386 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:31.768399 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:31.769164 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:31.769169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:31.769288 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:31.769934 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:31.769957 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:31.773813 (+ 3856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":288,"threads_started":1}
W20260928 03:05:31.774412 16213 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:31.763813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:42039 (local address 127.6.192.62:41481)
0928 03:05:31.764847 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:31.764851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:31.764870 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:05:31.764958 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:31.764972 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:31.765060 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:31.765188 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:31.766097 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:31.767015 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:31.768550 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:31.769013 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:31.769125 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:31.774092 (+ 4967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.1:42039: BlockingRecv error: recv got EOF from 127.6.192.1:42039 (error 108)
Metrics: {"server-negotiator.queue_time_us":921,"thread_start_us":745,"threads_started":1}
Sep 28 03:05:32 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:32 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:32.293902 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:32.284190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.2:53709)
0928 03:05:32.284757 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:32.284774 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:32.284881 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:32.285463 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:32.285468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:32.285935 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:32.286090 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:32.286099 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:32.287351 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:32.287356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:32.288534 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:32.288548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:32.289318 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:32.289323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:32.289492 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:32.290183 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:32.290202 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:32.293656 (+ 3454us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":350,"threads_started":1}
W20260928 03:05:32.294279 16215 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:32.284294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:53709 (local address 127.6.192.62:41481)
0928 03:05:32.285130 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:32.285136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:32.285154 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:05:32.285215 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:32.285227 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:32.285294 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:32.285389 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:32.286252 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:32.287179 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:32.288710 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:32.289154 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:32.289263 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:32.293963 (+ 4700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.2:53709: BlockingRecv error: recv got EOF from 127.6.192.2:53709 (error 108)
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":586,"threads_started":1}
Sep 28 03:05:32 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:32 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:32.776854 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:32.766671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.3:42271)
0928 03:05:32.767265 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:32.767283 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:32.767414 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:32.767859 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:32.767863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:32.768242 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:32.768402 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:32.768411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:32.769593 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:32.769598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:32.770549 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:32.770561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:32.771639 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:32.771646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:32.771861 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:32.772676 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:32.772702 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:32.776535 (+ 3833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":342,"threads_started":1}
W20260928 03:05:32.777029 16215 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:32.766897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:42271 (local address 127.6.192.62:41481)
0928 03:05:32.767120 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:32.767124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:32.767380 (+ 256us) server_negotiation.cc:407] Connection header received
0928 03:05:32.767583 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:32.767587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:32.767654 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:32.767753 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:32.768585 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:32.769375 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:32.770726 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:32.771369 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:32.771599 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:32.776759 (+ 5160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.3:42271: BlockingRecv error: recv got EOF from 127.6.192.3:42271 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 28 03:05:32 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:32 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:32.785619 16218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:32.775379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.1:56817)
0928 03:05:32.776250 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:32.776273 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:32.776416 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:32.776795 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:32.776800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:32.777417 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:32.777595 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:32.777604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:32.778872 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:32.778876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:32.780105 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:32.780120 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:32.780892 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:32.780896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:32.781016 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:32.781778 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:32.781796 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:32.785283 (+ 3487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":634,"threads_started":1}
W20260928 03:05:32.786082 16217 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:32.775480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:56817 (local address 127.6.192.62:41481)
0928 03:05:32.776175 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:32.776181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:32.776358 (+ 177us) server_negotiation.cc:407] Connection header received
0928 03:05:32.776552 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:32.776556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:32.776628 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:32.776738 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:32.777765 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:32.778726 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:32.780265 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:32.780747 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:32.780856 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:32.785726 (+ 4870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.1:56817: BlockingRecv error: recv got EOF from 127.6.192.1:56817 (error 108)
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":306,"threads_started":1}
Sep 28 03:05:33 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:33 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:33.308142 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:33.295063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.2:53861)
0928 03:05:33.295859 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:33.295883 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:33.296028 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:33.296458 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:33.296462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:33.296887 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:33.297077 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:33.297086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:33.298171 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:33.298175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:33.299240 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:33.299254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:33.300105 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:33.300112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:33.300295 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:33.304191 (+ 3896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:33.304222 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:33.307712 (+ 3490us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":464,"threads_started":1}
W20260928 03:05:33.308317 16220 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:33.295195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:53861 (local address 127.6.192.62:41481)
0928 03:05:33.296010 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:33.296014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:33.296036 (+ 22us) server_negotiation.cc:407] Connection header received
0928 03:05:33.296140 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:33.296144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:33.296209 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:33.296309 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:33.297227 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:33.298014 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:33.299429 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:33.299940 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:33.300086 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:33.308061 (+ 7975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.2:53861: BlockingRecv error: recv got EOF from 127.6.192.2:53861 (error 108)
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":579,"threads_started":1}
Sep 28 03:05:33 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:33 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 03:05:33.788842 16220 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:33.778319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:42779 (local address 127.6.192.62:41481)
0928 03:05:33.778543 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:33.778548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:33.779152 (+ 604us) server_negotiation.cc:407] Connection header received
0928 03:05:33.779333 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:33.779338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:33.779410 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:33.779598 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:33.780632 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:33.781503 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:33.782877 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:33.783336 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:33.783486 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:33.788536 (+ 5050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.3:42779: BlockingRecv error: recv got EOF from 127.6.192.3:42779 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 03:05:33.789577 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:33.778212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.3:42779)
0928 03:05:33.778995 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:33.779014 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:33.779156 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:33.779674 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:33.779678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:33.780153 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:33.780367 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:33.780381 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:33.781678 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:33.781682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:33.782703 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:33.782718 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:33.783601 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:33.783607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:33.783734 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:33.784477 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:33.784496 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:33.789228 (+ 4732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":505,"threads_started":1}
Sep 28 03:05:33 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:33 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:33.797820 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:33.786689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.1:50151)
0928 03:05:33.788780 (+ 2091us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:33.788805 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:33.788961 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:33.789424 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:33.789428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:33.789923 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:33.790082 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:33.790091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:33.791236 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:33.791240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:33.792345 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:33.792453 (+ 108us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:33.793248 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:33.793252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:33.793520 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:33.794162 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:33.794180 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:33.797565 (+ 3385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1961,"thread_start_us":1856,"threads_started":1}
W20260928 03:05:33.798092 16220 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:33.786720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:50151 (local address 127.6.192.62:41481)
0928 03:05:33.789100 (+ 2380us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:33.789106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:33.789126 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:05:33.789179 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:33.789183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:33.789267 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:33.789372 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:33.790224 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:33.791073 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:33.792625 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:33.793105 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:33.793209 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:33.797799 (+ 4590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.1:50151: BlockingRecv error: recv got EOF from 127.6.192.1:50151 (error 108)
Metrics: {"server-negotiator.queue_time_us":2254,"thread_start_us":2196,"threads_started":1}
Sep 28 03:05:34 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:34 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:34.318753 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:34.309197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.2:41451)
0928 03:05:34.310009 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:34.310041 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:34.310204 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:34.310664 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:34.310667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:34.311090 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:34.311263 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:34.311272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:34.312482 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:34.312486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:34.313434 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:34.313446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:34.314221 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:34.314225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:34.314363 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:34.315051 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:34.315070 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:34.318478 (+ 3408us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":321,"threads_started":1}
W20260928 03:05:34.319145 16225 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:34.309293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:41451 (local address 127.6.192.62:41481)
0928 03:05:34.310096 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:34.310105 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:34.310128 (+ 23us) server_negotiation.cc:407] Connection header received
0928 03:05:34.310373 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:34.310378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:34.310487 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:34.310585 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:34.311430 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:34.312328 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:34.313589 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:34.314079 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:34.314189 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:34.318764 (+ 4575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.2:41451: BlockingRecv error: recv got EOF from 127.6.192.2:41451 (error 108)
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":565,"threads_started":1}
Sep 28 03:05:34 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:34 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 03:05:34.803746 16225 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:34.790824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:38339 (local address 127.6.192.62:41481)
0928 03:05:34.791105 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:34.791113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:34.791385 (+ 272us) server_negotiation.cc:407] Connection header received
0928 03:05:34.791575 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:34.791579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:34.791647 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:34.791743 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:34.792708 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:34.793645 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:34.794932 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:34.795460 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:34.795612 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:34.803431 (+ 7819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.3:38339: BlockingRecv error: recv got EOF from 127.6.192.3:38339 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 03:05:34.803936 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:34.790737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.3:38339)
0928 03:05:34.791269 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:34.791287 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:34.791411 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:34.791794 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:34.791797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:34.792357 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:34.792536 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:34.792546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:34.793786 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:34.793790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:34.794784 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:34.794796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:34.795719 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:34.795727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:34.795974 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:34.796853 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:34.796882 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:34.803696 (+ 6814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":333,"threads_started":1}
Sep 28 03:05:34 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:34 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:34.810556 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:34.798926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.1:44993)
0928 03:05:34.801035 (+ 2109us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:34.801057 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:34.801197 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:34.802012 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:34.802019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:34.802573 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:34.802822 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:34.802838 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:34.804549 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:34.804553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:34.805498 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:34.805509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:34.806357 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:34.806361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:34.806541 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:34.807247 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:34.807264 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:34.810283 (+ 3019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1980,"thread_start_us":1824,"threads_started":1}
W20260928 03:05:34.810899 16228 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:34.799016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:44993 (local address 127.6.192.62:41481)
0928 03:05:34.799789 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:34.799793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:34.801334 (+ 1541us) server_negotiation.cc:407] Connection header received
0928 03:05:34.801422 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:34.801431 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:34.801547 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:34.801672 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:34.803039 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:34.804381 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:34.805647 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:34.806174 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:34.806284 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:34.810596 (+ 4312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.1:44993: BlockingRecv error: recv got EOF from 127.6.192.1:44993 (error 108)
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":517,"threads_started":1}
Sep 28 03:05:35 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:35 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:35.330106 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:35.319905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.2:35975)
0928 03:05:35.320645 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:35.320662 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:35.320786 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:35.321142 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:35.321145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:35.321661 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:35.321858 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:35.321867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:35.323066 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:35.323071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:35.324108 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:35.324122 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:35.325002 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:35.325007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:35.325135 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:35.326043 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:35.326064 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:35.329732 (+ 3668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":345,"threads_started":1}
W20260928 03:05:35.330314 16230 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:35.320076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:35975 (local address 127.6.192.62:41481)
0928 03:05:35.320580 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:35.320584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:35.320752 (+ 168us) server_negotiation.cc:407] Connection header received
0928 03:05:35.320931 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:35.320935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:35.320998 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:35.321091 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:35.322042 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:35.322882 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:35.324272 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:35.324790 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:35.324943 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:35.330049 (+ 5106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.2:35975: BlockingRecv error: recv got EOF from 127.6.192.2:35975 (error 108)
Metrics: {"server-negotiator.queue_time_us":348,"spinlock_wait_cycles":53504,"thread_start_us":252,"threads_started":1}
Sep 28 03:05:35 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:35 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:35.814509 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:35.805036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.3:46965)
0928 03:05:35.805694 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:35.805711 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:35.805830 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:35.806194 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:35.806198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:35.806611 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:35.806778 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:35.806787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:35.808024 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:35.808029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:35.809145 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:35.809160 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:35.809916 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:35.809921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:35.810056 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:35.810690 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:35.810708 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:35.814163 (+ 3455us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":402,"threads_started":1}
W20260928 03:05:35.814664 16230 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:35.805106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:46965 (local address 127.6.192.62:41481)
0928 03:05:35.805271 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:35.805275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:35.805805 (+ 530us) server_negotiation.cc:407] Connection header received
0928 03:05:35.805981 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:35.805985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:35.806048 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:35.806139 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:35.806946 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:35.807872 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:35.809330 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:35.809720 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:35.809824 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:35.814403 (+ 4579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.3:46965: BlockingRecv error: recv got EOF from 127.6.192.3:46965 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 03:05:35 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:35 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:35.821765 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:35.811790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.1:42605)
0928 03:05:35.812660 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:35.812679 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:35.812839 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:35.813174 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:35.813178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:35.813618 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:35.813797 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:35.813806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:35.815697 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:35.815704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:35.816824 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:35.816837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:35.817565 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:35.817571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:35.817756 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:35.818326 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:35.818342 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:35.821492 (+ 3150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":641,"threads_started":1}
W20260928 03:05:35.822064 16233 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:35.811885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:42605 (local address 127.6.192.62:41481)
0928 03:05:35.812632 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:35.812638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:35.812769 (+ 131us) server_negotiation.cc:407] Connection header received
0928 03:05:35.812964 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:35.812969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:35.813036 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:35.813143 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:35.813940 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:35.815551 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:35.816966 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:35.817413 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:35.817543 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:35.821751 (+ 4208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.1:42605: BlockingRecv error: recv got EOF from 127.6.192.1:42605 (error 108)
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":266,"threads_started":1}
Sep 28 03:05:36 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:36 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:36.341073 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:36.331314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.2:60147)
0928 03:05:36.331936 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:36.331959 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:36.332109 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:36.332653 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:36.332659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:36.333146 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:36.333387 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:36.333403 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:36.334764 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:36.334768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:36.336192 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:36.336206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:36.336979 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:36.336984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:36.337113 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:36.337746 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:36.337765 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:36.340834 (+ 3069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":324,"threads_started":1}
W20260928 03:05:36.341389 16235 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:36.331552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:60147 (local address 127.6.192.62:41481)
0928 03:05:36.332065 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:36.332070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:36.332091 (+ 21us) server_negotiation.cc:407] Connection header received
0928 03:05:36.332284 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:36.332291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:36.332374 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:36.332634 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:36.333581 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:36.334588 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:36.336362 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:36.336824 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:36.336933 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:36.341087 (+ 4154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.2:60147: BlockingRecv error: recv got EOF from 127.6.192.2:60147 (error 108)
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":285,"threads_started":1}
Sep 28 03:05:36 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:36 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.3@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:36.826516 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:36.815718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.3:59397)
0928 03:05:36.816283 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:36.816304 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:36.816450 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:36.817042 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:36.817047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:36.817568 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:36.817776 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:36.817787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:36.818932 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:36.818938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:36.820120 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:36.820134 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:36.820906 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:36.820912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:36.821055 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:36.821821 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:36.821841 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:36.826244 (+ 4403us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":342,"threads_started":1}
W20260928 03:05:36.826941 16235 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:36.815821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:59397 (local address 127.6.192.62:41481)
0928 03:05:36.816046 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:36.816051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:36.816428 (+ 377us) server_negotiation.cc:407] Connection header received
0928 03:05:36.816742 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:36.816746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:36.816832 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:36.816951 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:36.817959 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:36.818758 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:36.820307 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:36.820739 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:36.820846 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:36.826524 (+ 5678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.3:59397: BlockingRecv error: recv got EOF from 127.6.192.3:59397 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 03:05:36 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:36 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.1@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:36.833531 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:36.823351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.1:57591)
0928 03:05:36.824003 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:36.824024 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:36.824150 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:36.824987 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:36.824992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:36.825555 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:36.825828 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:36.825842 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:36.827360 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:36.827366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:36.828473 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:36.828490 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:36.829275 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:36.829280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:36.829411 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:36.830071 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:36.830089 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:36.833241 (+ 3152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":347,"threads_started":1}
W20260928 03:05:36.833993 16238 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:36.823538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:57591 (local address 127.6.192.62:41481)
0928 03:05:36.824573 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:36.824579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:36.824599 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:05:36.824650 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:36.824655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:36.824733 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:36.824855 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:36.826002 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:36.827198 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:36.828639 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:36.829076 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:36.829190 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:36.833553 (+ 4363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.1:57591: BlockingRecv error: recv got EOF from 127.6.192.1:57591 (error 108)
Metrics: {"server-negotiator.queue_time_us":904,"thread_start_us":544,"threads_started":1}
Sep 28 03:05:37 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 03:05:37 dist-test-slave-fjww krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.192.2@KRBTEST.COM for oryx/127.6.192.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 03:05:37.351559 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:37.342389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41481 (local address 127.6.192.2:48805)
0928 03:05:37.342943 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:37.342960 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:37.343079 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:37.343397 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:37.343400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:37.343917 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:37.344129 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:37.344139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:37.345252 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:37.345256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:37.346235 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:37.346248 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:37.347056 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:37.347061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:37.347184 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:37.347959 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:37.347981 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:37.351230 (+ 3249us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.192.62:41481: Server oryx/127.6.192.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":310,"threads_started":1}
W20260928 03:05:37.351881 16240 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:37.342492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:48805 (local address 127.6.192.62:41481)
0928 03:05:37.343032 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:37.343037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:37.343057 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:05:37.343190 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:37.343194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:37.343255 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:37.343355 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:37.344299 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:37.345090 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:37.346387 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:37.346864 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:37.346983 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:37.351562 (+ 4579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.192.2:48805: BlockingRecv error: recv got EOF from 127.6.192.2:48805 (error 108)
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":270,"threads_started":1}
I20260928 03:05:37.631151 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 15753
I20260928 03:05:37.645598 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:37.632601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53834 (local address 127.6.192.1:42937)
0928 03:05:37.633341 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:37.633349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:37.633378 (+ 29us) server_negotiation.cc:407] Connection header received
0928 03:05:37.633447 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:37.633457 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:37.633686 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:37.633854 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:37.635600 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:37.636778 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:37.638261 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:37.638872 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:37.639025 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:37.642104 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:37.642112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:37.642127 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:37.642168 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:37.645041 (+ 2873us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.192.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 03:05:37.645346 (+ 305us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53834: Request ticket server kudu/127.6.192.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":421,"threads_started":1}
W20260928 03:05:37.645745 16242 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53834: Request ticket server kudu/127.6.192.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 03:05:37.645998 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 15753: Not authorized: Client connection negotiation failed: client connection to 127.6.192.1:42937: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.192.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 03:05:37.646246 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 15753
I20260928 03:05:37.665349 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 15904
I20260928 03:05:37.680675 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:37.666664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52860 (local address 127.6.192.2:33971)
0928 03:05:37.667251 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:37.667259 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:37.667292 (+ 33us) server_negotiation.cc:407] Connection header received
0928 03:05:37.667363 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:37.667370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:37.667693 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:37.667924 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:37.669475 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:37.670664 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:37.672481 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:37.673156 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:37.673332 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:37.676617 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:37.676628 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:37.676643 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:37.676682 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:37.679904 (+ 3222us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.192.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 03:05:37.680303 (+ 399us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52860: Request ticket server kudu/127.6.192.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":310,"threads_started":1}
W20260928 03:05:37.680811 16243 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52860: Request ticket server kudu/127.6.192.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 03:05:37.680783 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 15904: Not authorized: Client connection negotiation failed: client connection to 127.6.192.2:33971: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.192.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 03:05:37.680931 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 15904
I20260928 03:05:37.694855 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 16036
I20260928 03:05:37.709049 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:37.696179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56488 (local address 127.6.192.3:42363)
0928 03:05:37.696775 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:37.696783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:37.696810 (+ 27us) server_negotiation.cc:407] Connection header received
0928 03:05:37.696869 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:37.696876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:37.697136 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:37.697299 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:37.698923 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:37.700202 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:37.701775 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:37.702526 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:37.702656 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:37.705538 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:37.705547 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:37.705562 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:37.705600 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:37.708503 (+ 2903us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.192.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 03:05:37.708806 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56488: Request ticket server kudu/127.6.192.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":320,"threads_started":1}
W20260928 03:05:37.709141 16244 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56488: Request ticket server kudu/127.6.192.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 03:05:37.709466 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 16036: Not authorized: Client connection negotiation failed: client connection to 127.6.192.3:42363: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.192.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 03:05:37.709619 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 16036
I20260928 03:05:37.726451 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 15684
I20260928 03:05:37.728212 15746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:37.855192 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 15684
2026-09-28T03:05:37Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21510 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T03:05:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:05:37Z Disabled control of system clock
I20260928 03:05:37.936540 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:40773
--webserver_interface=127.6.192.62
--webserver_port=0
--builtin_ntp_servers=127.6.192.20:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:40773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 03:05:38.279129 16249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:38.279547 16249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:38.279718 16249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:38.287269 16249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:05:38.287453 16249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:38.287575 16249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:38.287688 16249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:05:38.287765 16249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:05:38.297370 16249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:40773
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:40773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:38.298791 16249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:38.300773 16249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:38.313511 16254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:38.313730 16255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:38.314344 16257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:38.314998 16249 server_base.cc:1034] running on GCE node
I20260928 03:05:38.315975 16249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:38.317535 16249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:38.318771 16249 hybrid_clock.cc:648] HybridClock initialized: now 1790564738318723 us; error 38 us; skew 500 ppm
I20260928 03:05:38.319250 16249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:05:38.325580 16249 webserver.cc:460] Webserver started at http://127.6.192.62:43201/ using document root <none> and password file <none>
I20260928 03:05:38.326409 16249 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:38.326510 16249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:38.326875 16249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:38.330071 16249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "224a5246d87149de8f910dbf453a7a1e"
format_stamp: "Formatted at 2026-09-28 03:05:38 on dist-test-slave-fjww"
server_key: "c20fc0977972134466bbd84ea82924a8"
server_key_iv: "71fce5ecb23c37d93f7ed2ecad5a9a6d"
server_key_version: "encryptionkey@0"
I20260928 03:05:38.330947 16249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "224a5246d87149de8f910dbf453a7a1e"
format_stamp: "Formatted at 2026-09-28 03:05:38 on dist-test-slave-fjww"
server_key: "c20fc0977972134466bbd84ea82924a8"
server_key_iv: "71fce5ecb23c37d93f7ed2ecad5a9a6d"
server_key_version: "encryptionkey@0"
I20260928 03:05:38.336758 16249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 03:05:38.340567 16263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:38.342211 16249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 03:05:38.342440 16249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "224a5246d87149de8f910dbf453a7a1e"
format_stamp: "Formatted at 2026-09-28 03:05:38 on dist-test-slave-fjww"
server_key: "c20fc0977972134466bbd84ea82924a8"
server_key_iv: "71fce5ecb23c37d93f7ed2ecad5a9a6d"
server_key_version: "encryptionkey@0"
I20260928 03:05:38.342705 16249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:38.373840 16249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:38.374790 16249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:38.375116 16249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:38.396262 16249 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:40773
I20260928 03:05:38.396270 16314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:40773 every 8 connection(s)
I20260928 03:05:38.397704 16249 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:05:38.401894 16315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:38.402046 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 16249
I20260928 03:05:38.402261 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:05:38.403012 6912 external_mini_cluster.cc:1428] Setting key e825eabd5358396e4c91f26482030e82
I20260928 03:05:38.416868 16315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e: Bootstrap starting.
I20260928 03:05:38.418841 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:38.405122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.6.192.62:40773)
0928 03:05:38.406268 (+ 1146us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:38.406287 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:38.406330 (+ 43us) server_negotiation.cc:407] Connection header received
0928 03:05:38.408559 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:38.408587 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:38.409143 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:38.409567 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:38.410348 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:38.411911 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:38.413914 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:38.414752 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:38.414891 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:38.416261 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:38.416269 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:38.416285 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:05:38.416909 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:38.416993 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:05:38.417019 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:38.417158 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:38.417396 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 03:05:38.417819 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":415,"threads_started":1}
I20260928 03:05:38.420763 16315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:38.421990 16315 log.cc:824] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e: Log is configured to *not* fsync() on all Append() calls
I20260928 03:05:38.425228 16315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e: No bootstrap required, opened a new log
I20260928 03:05:38.433949 16315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224a5246d87149de8f910dbf453a7a1e" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40773 } }
I20260928 03:05:38.434379 16315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:38.434486 16315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 224a5246d87149de8f910dbf453a7a1e, State: Initialized, Role: FOLLOWER
I20260928 03:05:38.435119 16315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224a5246d87149de8f910dbf453a7a1e" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40773 } }
I20260928 03:05:38.435376 16315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:05:38.435573 16315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:05:38.435755 16315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:38.438120 16315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224a5246d87149de8f910dbf453a7a1e" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40773 } }
I20260928 03:05:38.438706 16315 leader_election.cc:302] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 224a5246d87149de8f910dbf453a7a1e; no voters:
I20260928 03:05:38.439314 16315 leader_election.cc:288] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:05:38.439569 16320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:05:38.440284 16320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e [term 1 LEADER]: Becoming Leader. State: Replica: 224a5246d87149de8f910dbf453a7a1e, State: Running, Role: LEADER
I20260928 03:05:38.440933 16320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224a5246d87149de8f910dbf453a7a1e" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40773 } }
I20260928 03:05:38.441673 16315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:05:38.443872 16322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 224a5246d87149de8f910dbf453a7a1e. Latest consensus state: current_term: 1 leader_uuid: "224a5246d87149de8f910dbf453a7a1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224a5246d87149de8f910dbf453a7a1e" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40773 } } }
I20260928 03:05:38.444106 16321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "224a5246d87149de8f910dbf453a7a1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224a5246d87149de8f910dbf453a7a1e" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40773 } } }
I20260928 03:05:38.444365 16322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e [sys.catalog]: This master's current role is: LEADER
I20260928 03:05:38.444445 16321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e [sys.catalog]: This master's current role is: LEADER
I20260928 03:05:38.446265 16328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:05:38.452582 16328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:05:38.462491 16328 catalog_manager.cc:1347] Generated new cluster ID: c2bef20b21e14d0e8875de91c91fb80c
I20260928 03:05:38.462649 16328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:05:38.475546 16328 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:05:38.477130 16328 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:05:38.485096 16328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 224a5246d87149de8f910dbf453a7a1e: Generated new TSK 0
I20260928 03:05:38.486099 16328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:05:38.504261 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:40773
--builtin_ntp_servers=127.6.192.20:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 03:05:38.857870 16339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:38.858196 16339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:38.858361 16339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:38.866493 16339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:38.866618 16339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:38.866813 16339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:05:38.878425 16339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:40773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:38.879902 16339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:38.881901 16339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:38.896857 16344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:38.896829 16347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:38.896780 16345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:38.898228 16339 server_base.cc:1034] running on GCE node
I20260928 03:05:38.898910 16339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:38.900115 16339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:38.901376 16339 hybrid_clock.cc:648] HybridClock initialized: now 1790564738901243 us; error 113 us; skew 500 ppm
I20260928 03:05:38.901774 16339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:05:38.904737 16339 webserver.cc:460] Webserver started at http://127.6.192.1:46799/ using document root <none> and password file <none>
I20260928 03:05:38.905555 16339 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:38.905669 16339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:38.906039 16339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:38.910030 16339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "054a52e23dff4d8aa8529a81e60151d8"
format_stamp: "Formatted at 2026-09-28 03:05:38 on dist-test-slave-fjww"
server_key: "3b1e3d45a24da92fa5ae90b92bb19395"
server_key_iv: "3284d3488f7ad2468fd1fb837dcc3cfc"
server_key_version: "encryptionkey@0"
I20260928 03:05:38.911307 16339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "054a52e23dff4d8aa8529a81e60151d8"
format_stamp: "Formatted at 2026-09-28 03:05:38 on dist-test-slave-fjww"
server_key: "3b1e3d45a24da92fa5ae90b92bb19395"
server_key_iv: "3284d3488f7ad2468fd1fb837dcc3cfc"
server_key_version: "encryptionkey@0"
I20260928 03:05:38.917419 16339 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260928 03:05:38.922163 16353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:38.923905 16339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:05:38.924125 16339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "054a52e23dff4d8aa8529a81e60151d8"
format_stamp: "Formatted at 2026-09-28 03:05:38 on dist-test-slave-fjww"
server_key: "3b1e3d45a24da92fa5ae90b92bb19395"
server_key_iv: "3284d3488f7ad2468fd1fb837dcc3cfc"
server_key_version: "encryptionkey@0"
I20260928 03:05:38.924367 16339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:38.953133 16339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:38.954191 16339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:38.954499 16339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:38.955608 16339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:05:38.957127 16339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:38.957216 16339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:38.957343 16339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:38.957439 16339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:38.994175 16339 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:36331
I20260928 03:05:38.994292 16465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:36331 every 8 connection(s)
I20260928 03:05:38.995651 16339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:05:39.001973 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 16339
I20260928 03:05:39.002203 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:05:39.003098 6912 external_mini_cluster.cc:1428] Setting key 1134176f886783058f84ba93019bb9bf
I20260928 03:05:39.006395 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:40773
--builtin_ntp_servers=127.6.192.20:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 03:05:39.011240 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:38.998650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:33777 (local address 127.6.192.62:40773)
0928 03:05:38.999159 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:38.999165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:39.000633 (+ 1468us) server_negotiation.cc:407] Connection header received
0928 03:05:39.003326 (+ 2693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:39.003331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:39.003422 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:39.003603 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:39.005105 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:39.005880 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:39.007860 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:39.008283 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:39.008432 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:39.009990 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:39.010000 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:39.010008 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:05:39.010039 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:39.010098 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:05:39.010119 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:39.010232 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:39.010600 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 03:05:39.010730 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":236,"threads_started":1}
I20260928 03:05:39.012157 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:38.999276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:40773 (local address 127.6.192.1:33777)
0928 03:05:39.000445 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:39.000503 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:39.002954 (+ 2451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:39.003905 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:39.003916 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:39.004028 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:39.004868 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:39.004897 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:39.006081 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:39.006086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:39.007688 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:39.007704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:39.008554 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:39.008564 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:39.008798 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:39.009547 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:05:39.009549 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:39.009594 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:05:39.009721 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:39.010260 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:39.010270 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:39.010277 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 03:05:39.010558 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 03:05:39.010905 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":374,"threads_started":1}
I20260928 03:05:39.014179 16466 heartbeater.cc:347] Connected to a master server at 127.6.192.62:40773
I20260928 03:05:39.014533 16466 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:39.015376 16466 heartbeater.cc:511] Master 127.6.192.62:40773 requested a full tablet report, sending...
I20260928 03:05:39.017781 16280 ts_manager.cc:194] Registered new tserver with Master: 054a52e23dff4d8aa8529a81e60151d8 (127.6.192.1:36331)
I20260928 03:05:39.019279 16280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.192.1:33777
W20260928 03:05:39.347944 16470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:39.348253 16470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:39.348417 16470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:39.356123 16470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:39.356242 16470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:39.356426 16470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:05:39.365757 16470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:40773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:39.367161 16470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:39.369038 16470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:39.383059 16475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:39.383132 16476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:39.383138 16478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:39.384298 16470 server_base.cc:1034] running on GCE node
I20260928 03:05:39.384938 16470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:39.385950 16470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:39.387174 16470 hybrid_clock.cc:648] HybridClock initialized: now 1790564739387119 us; error 44 us; skew 500 ppm
I20260928 03:05:39.387537 16470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:05:39.389953 16470 webserver.cc:460] Webserver started at http://127.6.192.2:36543/ using document root <none> and password file <none>
I20260928 03:05:39.390656 16470 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:39.390748 16470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:39.391029 16470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:39.394706 16470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "28d18923e1d74dafa045580152b43f20"
format_stamp: "Formatted at 2026-09-28 03:05:39 on dist-test-slave-fjww"
server_key: "88197817da6ac5526bd9edf4d74825db"
server_key_iv: "84341e777df8b536283f7ea96d96e4cc"
server_key_version: "encryptionkey@0"
I20260928 03:05:39.396019 16470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "28d18923e1d74dafa045580152b43f20"
format_stamp: "Formatted at 2026-09-28 03:05:39 on dist-test-slave-fjww"
server_key: "88197817da6ac5526bd9edf4d74825db"
server_key_iv: "84341e777df8b536283f7ea96d96e4cc"
server_key_version: "encryptionkey@0"
I20260928 03:05:39.402475 16470 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 03:05:39.406659 16484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:39.408332 16470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:05:39.408545 16470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "28d18923e1d74dafa045580152b43f20"
format_stamp: "Formatted at 2026-09-28 03:05:39 on dist-test-slave-fjww"
server_key: "88197817da6ac5526bd9edf4d74825db"
server_key_iv: "84341e777df8b536283f7ea96d96e4cc"
server_key_version: "encryptionkey@0"
I20260928 03:05:39.408799 16470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:39.422729 16470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:39.423790 16470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:39.424101 16470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:39.425139 16470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:05:39.426690 16470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:39.426820 16470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:39.426952 16470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:39.427040 16470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:39.465236 16470 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:36301
I20260928 03:05:39.465291 16596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:36301 every 8 connection(s)
I20260928 03:05:39.466717 16470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:05:39.474965 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 16470
I20260928 03:05:39.475155 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:05:39.475708 6912 external_mini_cluster.cc:1428] Setting key a233523df040ef7841f3c7defd620ff1
I20260928 03:05:39.479123 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:40773
--builtin_ntp_servers=127.6.192.20:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 03:05:39.482725 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:39.469746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:57579 (local address 127.6.192.62:40773)
0928 03:05:39.469957 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:39.469962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:39.471848 (+ 1886us) server_negotiation.cc:407] Connection header received
0928 03:05:39.474083 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:39.474090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:39.474228 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:39.474372 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:39.475964 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:39.476948 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:39.479111 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:39.479928 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:39.480184 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:39.481626 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:39.481635 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:39.481644 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:05:39.481672 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:39.481717 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:05:39.481738 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:39.481842 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:39.482145 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 03:05:39.482271 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 03:05:39.483791 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:39.470314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:40773 (local address 127.6.192.2:57579)
0928 03:05:39.471396 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:39.471602 (+ 206us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:39.473564 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:39.474729 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:39.474742 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:39.474892 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:39.475737 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:39.475758 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:39.477125 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:39.477130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:39.478929 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:39.478942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:39.480245 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:39.480250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:39.480499 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:39.481260 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:05:39.481263 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:39.481313 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:05:39.481403 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:39.481873 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:39.481882 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:39.481888 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 03:05:39.482099 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 03:05:39.482445 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":316,"threads_started":1}
I20260928 03:05:39.485406 16597 heartbeater.cc:347] Connected to a master server at 127.6.192.62:40773
I20260928 03:05:39.485733 16597 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:39.486552 16597 heartbeater.cc:511] Master 127.6.192.62:40773 requested a full tablet report, sending...
I20260928 03:05:39.488058 16280 ts_manager.cc:194] Registered new tserver with Master: 28d18923e1d74dafa045580152b43f20 (127.6.192.2:36301)
I20260928 03:05:39.488950 16280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.192.2:57579
W20260928 03:05:39.829438 16600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:39.829770 16600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:39.829941 16600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:39.838827 16600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:39.838966 16600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:39.839170 16600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:05:39.850129 16600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:40773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:39.851630 16600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:39.853648 16600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:39.869203 16605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:39.869235 16606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:39.869261 16608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:39.870010 16600 server_base.cc:1034] running on GCE node
I20260928 03:05:39.870707 16600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:39.871917 16600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:39.873173 16600 hybrid_clock.cc:648] HybridClock initialized: now 1790564739873116 us; error 53 us; skew 500 ppm
I20260928 03:05:39.873651 16600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:05:39.876478 16600 webserver.cc:460] Webserver started at http://127.6.192.3:39765/ using document root <none> and password file <none>
I20260928 03:05:39.877234 16600 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:39.877375 16600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:39.877730 16600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:39.880810 16600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "a665c87e98354c68bce3a6e0b0dc3aad"
format_stamp: "Formatted at 2026-09-28 03:05:39 on dist-test-slave-fjww"
server_key: "51d0c02d3935a3d78ed2d1ad28726f6c"
server_key_iv: "18c8c8019667f0bd7234761ed53a0440"
server_key_version: "encryptionkey@0"
I20260928 03:05:39.881680 16600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "a665c87e98354c68bce3a6e0b0dc3aad"
format_stamp: "Formatted at 2026-09-28 03:05:39 on dist-test-slave-fjww"
server_key: "51d0c02d3935a3d78ed2d1ad28726f6c"
server_key_iv: "18c8c8019667f0bd7234761ed53a0440"
server_key_version: "encryptionkey@0"
I20260928 03:05:39.887331 16600 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 03:05:39.891227 16614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:39.893115 16600 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:05:39.893337 16600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "a665c87e98354c68bce3a6e0b0dc3aad"
format_stamp: "Formatted at 2026-09-28 03:05:39 on dist-test-slave-fjww"
server_key: "51d0c02d3935a3d78ed2d1ad28726f6c"
server_key_iv: "18c8c8019667f0bd7234761ed53a0440"
server_key_version: "encryptionkey@0"
I20260928 03:05:39.893590 16600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:39.910118 16600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:39.911312 16600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:39.911701 16600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:39.912995 16600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:05:39.914902 16600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:39.915046 16600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:39.915196 16600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:39.915295 16600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:39.949601 16600 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:40523
I20260928 03:05:39.949648 16726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:40523 every 8 connection(s)
I20260928 03:05:39.950981 16600 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:05:39.957072 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 16600
I20260928 03:05:39.957244 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:05:39.957675 6912 external_mini_cluster.cc:1428] Setting key 7bfaea07131f89fda4f8fb8702584546
I20260928 03:05:39.964843 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:39.953744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:50609 (local address 127.6.192.62:40773)
0928 03:05:39.953947 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:39.953952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:39.955408 (+ 1456us) server_negotiation.cc:407] Connection header received
0928 03:05:39.957324 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:39.957330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:39.957430 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:39.957545 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:39.958937 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:39.960254 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:39.962103 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:39.962487 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:39.962599 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:39.964142 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:39.964147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:39.964150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:05:39.964170 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:39.964201 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:05:39.964213 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:39.964292 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:39.964489 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 03:05:39.964595 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 03:05:39.965904 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:39.954319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:40773 (local address 127.6.192.3:50609)
0928 03:05:39.955242 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:39.955278 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:39.957016 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:39.957835 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:39.957847 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:39.957956 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:39.958730 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:39.958758 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:39.960417 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:39.960422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:39.961954 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:39.961969 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:39.962752 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:39.962758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:39.963003 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:39.963821 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:05:39.963823 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:39.963866 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:05:39.963951 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:39.964302 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:39.964309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:39.964315 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 03:05:39.964461 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 03:05:39.964814 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":309,"threads_started":1}
I20260928 03:05:39.967519 16727 heartbeater.cc:347] Connected to a master server at 127.6.192.62:40773
I20260928 03:05:39.967856 16727 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:39.968726 16727 heartbeater.cc:511] Master 127.6.192.62:40773 requested a full tablet report, sending...
I20260928 03:05:39.970227 16280 ts_manager.cc:194] Registered new tserver with Master: a665c87e98354c68bce3a6e0b0dc3aad (127.6.192.3:40523)
I20260928 03:05:39.970995 16280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.192.3:50609
I20260928 03:05:39.973987 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 03:05:39.980748 16468 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:39.978936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47164 (local address 127.6.192.62:40773)
0928 03:05:39.979174 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:39.979178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:39.979737 (+ 559us) server_negotiation.cc:407] Connection header received
0928 03:05:39.979896 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:39.979900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:39.979979 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:39.980071 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:39.980557 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47164: BlockingRecv error: recv got EOF from 127.0.0.1:47164 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 03:05:39.982052 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 16339
I20260928 03:05:39.990347 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:39.983032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51670 (local address 127.6.192.1:36331)
0928 03:05:39.983600 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:39.983607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:39.983638 (+ 31us) server_negotiation.cc:407] Connection header received
0928 03:05:39.983706 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:39.983712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:39.983948 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:39.984097 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:39.984525 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:39.985527 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:39.986968 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:39.987658 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:39.987764 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:39.988855 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:39.988863 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:39.988887 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:05:39.989523 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:39.989573 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:05:39.989593 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:39.989691 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:39.989900 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 03:05:39.990029 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":258,"threads_started":1}
I20260928 03:05:39.991779 16461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:40.075330 16466 heartbeater.cc:503] Master 127.6.192.62:40773 was elected leader, sending a full tablet report...
I20260928 03:05:40.083714 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 16339
I20260928 03:05:40.100903 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 16470
I20260928 03:05:40.110975 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:40.101968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.6.192.2:36301)
0928 03:05:40.102704 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:40.102717 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:40.102754 (+ 37us) server_negotiation.cc:407] Connection header received
0928 03:05:40.102855 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:40.102868 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:40.103240 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:40.103400 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:40.104068 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:40.105215 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:40.106766 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:40.107407 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:40.107609 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:40.108841 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:40.108852 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:40.108874 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:05:40.109751 (+ 877us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:40.109865 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:05:40.109894 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:40.110030 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:40.110259 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 03:05:40.110566 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":300,"threads_started":1}
I20260928 03:05:40.112795 16592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:40.196192 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 16470
I20260928 03:05:40.212984 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 16600
I20260928 03:05:40.222652 16740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:40.214003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33418 (local address 127.6.192.3:40523)
0928 03:05:40.214721 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:40.214728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:40.214761 (+ 33us) server_negotiation.cc:407] Connection header received
0928 03:05:40.214833 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:40.214840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:40.215069 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:40.215214 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:40.215875 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:40.217111 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:40.218526 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:40.219199 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:40.219316 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:40.220863 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:40.220872 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:40.220888 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:05:40.221597 (+ 709us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:40.221655 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:05:40.221677 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:40.221783 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:40.221972 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 03:05:40.222176 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":325,"threads_started":1}
I20260928 03:05:40.224222 16722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:40.311194 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 16600
I20260928 03:05:40.327787 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 16249
I20260928 03:05:40.329255 16310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:40.444828 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 16249
2026-09-28T03:05:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2585 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T03:05:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:05:40Z Disabled control of system clock
I20260928 03:05:40.524544 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:34319
--webserver_interface=127.6.192.62
--webserver_port=0
--builtin_ntp_servers=127.6.192.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:34319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 03:05:40.874907 16746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:40.875259 16746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:40.875350 16746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:40.882790 16746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:05:40.882915 16746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:40.882952 16746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:40.883004 16746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:05:40.883049 16746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:05:40.892609 16746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:34319
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:34319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:40.893922 16746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:40.895772 16746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:40.910375 16752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:40.910375 16754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:40.910403 16751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:40.911146 16746 server_base.cc:1034] running on GCE node
I20260928 03:05:40.912182 16746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:40.913805 16746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:40.915091 16746 hybrid_clock.cc:648] HybridClock initialized: now 1790564740915045 us; error 52 us; skew 500 ppm
I20260928 03:05:40.915591 16746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:05:40.918522 16746 webserver.cc:460] Webserver started at http://127.6.192.62:35293/ using document root <none> and password file <none>
I20260928 03:05:40.919497 16746 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:40.919653 16746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:40.920034 16746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:40.925076 16746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "4b1421fc1dea497cafd14318c7770675"
format_stamp: "Formatted at 2026-09-28 03:05:40 on dist-test-slave-fjww"
server_key: "318b32b8b096041779a8eb9369c3e7b0"
server_key_iv: "d30cb3a8dd7cfc2364835b8a87a03c0d"
server_key_version: "encryptionkey@0"
I20260928 03:05:40.926075 16746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "4b1421fc1dea497cafd14318c7770675"
format_stamp: "Formatted at 2026-09-28 03:05:40 on dist-test-slave-fjww"
server_key: "318b32b8b096041779a8eb9369c3e7b0"
server_key_iv: "d30cb3a8dd7cfc2364835b8a87a03c0d"
server_key_version: "encryptionkey@0"
I20260928 03:05:40.932437 16746 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260928 03:05:40.936390 16760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:40.938216 16746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 03:05:40.938467 16746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "4b1421fc1dea497cafd14318c7770675"
format_stamp: "Formatted at 2026-09-28 03:05:40 on dist-test-slave-fjww"
server_key: "318b32b8b096041779a8eb9369c3e7b0"
server_key_iv: "d30cb3a8dd7cfc2364835b8a87a03c0d"
server_key_version: "encryptionkey@0"
I20260928 03:05:40.938736 16746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:40.945547 16746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:40.946462 16746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:40.946805 16746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:40.968262 16746 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:34319
I20260928 03:05:40.968300 16811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:34319 every 8 connection(s)
I20260928 03:05:40.969794 16746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:05:40.970383 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 16746
I20260928 03:05:40.970713 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:05:40.971719 6912 external_mini_cluster.cc:1428] Setting key 1ba118929abc2e3d5382c1b943e9cd9a
I20260928 03:05:40.975018 16812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:40.982892 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:40.974020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.6.192.62:34319)
0928 03:05:40.976602 (+ 2582us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:40.976622 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:40.976671 (+ 49us) server_negotiation.cc:407] Connection header received
0928 03:05:40.978434 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:40.978460 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:40.978958 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:40.979368 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:40.980736 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:40.980744 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:40.980759 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:05:40.980780 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:40.980904 (+ 124us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:05:40.980929 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:40.981172 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:40.981293 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 03:05:40.981668 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2091,"thread_start_us":1823,"threads_started":1}
I20260928 03:05:40.990090 16812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675: Bootstrap starting.
I20260928 03:05:40.993852 16812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:40.995110 16812 log.cc:824] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675: Log is configured to *not* fsync() on all Append() calls
I20260928 03:05:40.998549 16812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675: No bootstrap required, opened a new log
I20260928 03:05:41.007261 16812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1421fc1dea497cafd14318c7770675" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34319 } }
I20260928 03:05:41.007819 16812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:41.007949 16812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b1421fc1dea497cafd14318c7770675, State: Initialized, Role: FOLLOWER
I20260928 03:05:41.008685 16812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1421fc1dea497cafd14318c7770675" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34319 } }
I20260928 03:05:41.008980 16812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:05:41.009147 16812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:05:41.009351 16812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:41.011953 16812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1421fc1dea497cafd14318c7770675" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34319 } }
I20260928 03:05:41.012566 16812 leader_election.cc:302] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b1421fc1dea497cafd14318c7770675; no voters:
I20260928 03:05:41.013355 16812 leader_election.cc:288] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:05:41.013499 16817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:05:41.014287 16817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675 [term 1 LEADER]: Becoming Leader. State: Replica: 4b1421fc1dea497cafd14318c7770675, State: Running, Role: LEADER
I20260928 03:05:41.015110 16817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1421fc1dea497cafd14318c7770675" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34319 } }
I20260928 03:05:41.016198 16812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:05:41.018779 16819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b1421fc1dea497cafd14318c7770675. Latest consensus state: current_term: 1 leader_uuid: "4b1421fc1dea497cafd14318c7770675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1421fc1dea497cafd14318c7770675" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34319 } } }
I20260928 03:05:41.019230 16819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675 [sys.catalog]: This master's current role is: LEADER
I20260928 03:05:41.021466 16825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:05:41.021373 16818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b1421fc1dea497cafd14318c7770675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1421fc1dea497cafd14318c7770675" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34319 } } }
I20260928 03:05:41.021770 16818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675 [sys.catalog]: This master's current role is: LEADER
I20260928 03:05:41.026423 16825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:05:41.036180 16825 catalog_manager.cc:1347] Generated new cluster ID: d21ad5eacc0a4c378f797a615fb41092
I20260928 03:05:41.036340 16825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:05:41.065166 16825 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:05:41.065769 16825 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:05:41.076239 16825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b1421fc1dea497cafd14318c7770675: Generated new TSK 0
I20260928 03:05:41.077509 16825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:05:41.086566 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34319
--builtin_ntp_servers=127.6.192.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 03:05:41.438221 16836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:41.438565 16836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:41.438730 16836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:41.446610 16836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:41.446779 16836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:41.446959 16836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:05:41.456836 16836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:41.458206 16836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:41.460085 16836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:41.473922 16844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:41.473970 16842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:41.474506 16841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:41.474797 16836 server_base.cc:1034] running on GCE node
I20260928 03:05:41.475490 16836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:41.476815 16836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:41.478067 16836 hybrid_clock.cc:648] HybridClock initialized: now 1790564741478033 us; error 41 us; skew 500 ppm
I20260928 03:05:41.478420 16836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:05:41.481050 16836 webserver.cc:460] Webserver started at http://127.6.192.1:42671/ using document root <none> and password file <none>
I20260928 03:05:41.481808 16836 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:41.481943 16836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:41.482281 16836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:41.485260 16836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "84e05034091c4cf78373f98a21fd9920"
format_stamp: "Formatted at 2026-09-28 03:05:41 on dist-test-slave-fjww"
server_key: "cc93a96f9286f23b0d5ad992630e9366"
server_key_iv: "0a8e6341a300284ca50f757b36aeee8d"
server_key_version: "encryptionkey@0"
I20260928 03:05:41.486146 16836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "84e05034091c4cf78373f98a21fd9920"
format_stamp: "Formatted at 2026-09-28 03:05:41 on dist-test-slave-fjww"
server_key: "cc93a96f9286f23b0d5ad992630e9366"
server_key_iv: "0a8e6341a300284ca50f757b36aeee8d"
server_key_version: "encryptionkey@0"
I20260928 03:05:41.491807 16836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 03:05:41.495999 16850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:41.497666 16836 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 03:05:41.497856 16836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "84e05034091c4cf78373f98a21fd9920"
format_stamp: "Formatted at 2026-09-28 03:05:41 on dist-test-slave-fjww"
server_key: "cc93a96f9286f23b0d5ad992630e9366"
server_key_iv: "0a8e6341a300284ca50f757b36aeee8d"
server_key_version: "encryptionkey@0"
I20260928 03:05:41.498137 16836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:41.504104 16836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:41.504922 16836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:41.505208 16836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:41.506129 16836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:05:41.508072 16836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:41.508209 16836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:41.508337 16836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:41.508423 16836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:41.544767 16836 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:38223
I20260928 03:05:41.544822 16962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:38223 every 8 connection(s)
I20260928 03:05:41.546110 16836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:05:41.554318 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 16836
I20260928 03:05:41.554571 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:05:41.555092 16965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:41.549009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:46949 (local address 127.6.192.62:34319)
0928 03:05:41.549483 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:41.549488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:41.550797 (+ 1309us) server_negotiation.cc:407] Connection header received
0928 03:05:41.552620 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:41.552625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:41.552708 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:41.552830 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:41.554175 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:41.554180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:41.554185 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:05:41.554212 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:41.554256 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:05:41.554267 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:41.554407 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:41.554709 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 03:05:41.554783 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":258,"threads_started":1}
I20260928 03:05:41.555231 6912 external_mini_cluster.cc:1428] Setting key e6b98345b8acd8112770f3b84924b94c
I20260928 03:05:41.556154 16966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:41.549608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34319 (local address 127.6.192.1:46949)
0928 03:05:41.550627 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:41.550675 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:41.552328 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:41.553092 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:41.553100 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:41.553197 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:41.553870 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:05:41.553872 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:41.553910 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:05:41.554000 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:41.554364 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:41.554375 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:41.554383 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 03:05:41.554794 (+ 411us) client_negotiation.cc:240] Negotiation successful
0928 03:05:41.555067 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":289,"threads_started":1}
I20260928 03:05:41.557876 16963 heartbeater.cc:347] Connected to a master server at 127.6.192.62:34319
I20260928 03:05:41.558221 16963 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:41.558732 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34319
--builtin_ntp_servers=127.6.192.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 03:05:41.558961 16963 heartbeater.cc:511] Master 127.6.192.62:34319 requested a full tablet report, sending...
I20260928 03:05:41.561401 16776 ts_manager.cc:194] Registered new tserver with Master: 84e05034091c4cf78373f98a21fd9920 (127.6.192.1:38223)
W20260928 03:05:41.902318 16967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:41.902784 16967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:41.903019 16967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:41.911782 16967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:41.911962 16967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:41.912144 16967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:05:41.923395 16967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:41.925060 16967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:41.927073 16967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:41.942272 16973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:41.942304 16972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:41.942235 16975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:41.943228 16967 server_base.cc:1034] running on GCE node
I20260928 03:05:41.944290 16967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:41.945345 16967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:41.946660 16967 hybrid_clock.cc:648] HybridClock initialized: now 1790564741946604 us; error 50 us; skew 500 ppm
I20260928 03:05:41.947043 16967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:05:41.949883 16967 webserver.cc:460] Webserver started at http://127.6.192.2:35749/ using document root <none> and password file <none>
I20260928 03:05:41.950655 16967 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:41.950793 16967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:41.951159 16967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:41.954192 16967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "f228800be2d94d309437df2c79325219"
format_stamp: "Formatted at 2026-09-28 03:05:41 on dist-test-slave-fjww"
server_key: "e108d0bd18e5a41c93d38a31224e3d25"
server_key_iv: "8e59e8918f38069c9cd30a432af28461"
server_key_version: "encryptionkey@0"
I20260928 03:05:41.955039 16967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "f228800be2d94d309437df2c79325219"
format_stamp: "Formatted at 2026-09-28 03:05:41 on dist-test-slave-fjww"
server_key: "e108d0bd18e5a41c93d38a31224e3d25"
server_key_iv: "8e59e8918f38069c9cd30a432af28461"
server_key_version: "encryptionkey@0"
I20260928 03:05:41.960741 16967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 03:05:41.965023 16981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:41.966770 16967 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260928 03:05:41.967080 16967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "f228800be2d94d309437df2c79325219"
format_stamp: "Formatted at 2026-09-28 03:05:41 on dist-test-slave-fjww"
server_key: "e108d0bd18e5a41c93d38a31224e3d25"
server_key_iv: "8e59e8918f38069c9cd30a432af28461"
server_key_version: "encryptionkey@0"
I20260928 03:05:41.967338 16967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:41.973726 16967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:41.974624 16967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:41.974927 16967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:41.976109 16967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:05:41.977663 16967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:41.977792 16967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:41.977926 16967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:41.978021 16967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:42.012743 16967 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:45773
I20260928 03:05:42.012796 17094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:45773 every 8 connection(s)
I20260928 03:05:42.014271 16967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:05:42.024103 16965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:42.017385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:52855 (local address 127.6.192.62:34319)
0928 03:05:42.017694 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:42.017701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:42.019302 (+ 1601us) server_negotiation.cc:407] Connection header received
0928 03:05:42.021264 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:42.021269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:42.021342 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:42.021441 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:42.023084 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:42.023088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:42.023091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:05:42.023109 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:42.023141 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:05:42.023152 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:42.023258 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:42.023619 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 03:05:42.023726 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260928 03:05:42.024168 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 16967
I20260928 03:05:42.024371 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:05:42.024940 6912 external_mini_cluster.cc:1428] Setting key cb22fa9732cf8e36b9f9a01b0864170f
I20260928 03:05:42.025338 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:42.017897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34319 (local address 127.6.192.2:52855)
0928 03:05:42.019128 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:42.019170 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:42.020971 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:42.021782 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:42.021791 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:42.021910 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:42.022723 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:05:42.022726 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:42.022768 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:05:42.022872 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:42.023262 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:42.023271 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:42.023278 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 03:05:42.023714 (+ 436us) client_negotiation.cc:240] Negotiation successful
0928 03:05:42.024028 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":304,"threads_started":1}
I20260928 03:05:42.027200 17095 heartbeater.cc:347] Connected to a master server at 127.6.192.62:34319
I20260928 03:05:42.027644 17095 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:42.028290 17095 heartbeater.cc:511] Master 127.6.192.62:34319 requested a full tablet report, sending...
I20260928 03:05:42.028499 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34319
--builtin_ntp_servers=127.6.192.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 03:05:42.029985 16776 ts_manager.cc:194] Registered new tserver with Master: f228800be2d94d309437df2c79325219 (127.6.192.2:45773)
W20260928 03:05:42.369923 17098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:42.370241 17098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:42.370334 17098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:42.378111 17098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:42.378237 17098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:42.378355 17098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:05:42.387868 17098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:42.389153 17098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:42.390946 17098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:42.404393 17103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:42.404548 17104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:42.404803 17106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:42.405483 17098 server_base.cc:1034] running on GCE node
I20260928 03:05:42.406338 17098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:42.407552 17098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:42.408785 17098 hybrid_clock.cc:648] HybridClock initialized: now 1790564742408670 us; error 113 us; skew 500 ppm
I20260928 03:05:42.409149 17098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:05:42.411679 17098 webserver.cc:460] Webserver started at http://127.6.192.3:37751/ using document root <none> and password file <none>
I20260928 03:05:42.412412 17098 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:42.412505 17098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:42.412882 17098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:42.415932 17098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "c3b9ed00864a43969bc42c950840064f"
format_stamp: "Formatted at 2026-09-28 03:05:42 on dist-test-slave-fjww"
server_key: "c9b29229c6503079b0c35bec098efabd"
server_key_iv: "78bf31843b2caa6f850e37885ec1c41e"
server_key_version: "encryptionkey@0"
I20260928 03:05:42.416831 17098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "c3b9ed00864a43969bc42c950840064f"
format_stamp: "Formatted at 2026-09-28 03:05:42 on dist-test-slave-fjww"
server_key: "c9b29229c6503079b0c35bec098efabd"
server_key_iv: "78bf31843b2caa6f850e37885ec1c41e"
server_key_version: "encryptionkey@0"
I20260928 03:05:42.422595 17098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:05:42.426818 17112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:42.428478 17098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:05:42.428678 17098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "c3b9ed00864a43969bc42c950840064f"
format_stamp: "Formatted at 2026-09-28 03:05:42 on dist-test-slave-fjww"
server_key: "c9b29229c6503079b0c35bec098efabd"
server_key_iv: "78bf31843b2caa6f850e37885ec1c41e"
server_key_version: "encryptionkey@0"
I20260928 03:05:42.428933 17098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:42.435505 17098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:42.436405 17098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:42.436697 17098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:42.437716 17098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:05:42.439266 17098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:42.439355 17098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:42.439607 17098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:42.439687 17098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:42.476020 17098 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:38445
I20260928 03:05:42.476075 17224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:38445 every 8 connection(s)
I20260928 03:05:42.477459 17098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:05:42.483433 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 17098
I20260928 03:05:42.483695 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:05:42.484261 6912 external_mini_cluster.cc:1428] Setting key e398b803ec7a1a539ae971c623a4d097
I20260928 03:05:42.486179 16965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:42.480200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:53575 (local address 127.6.192.62:34319)
0928 03:05:42.480374 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:42.480378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:42.481817 (+ 1439us) server_negotiation.cc:407] Connection header received
0928 03:05:42.483715 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:42.483722 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:42.483799 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:42.483924 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:42.485315 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:42.485319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:42.485322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:05:42.485339 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:42.485378 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:05:42.485391 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:42.485540 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:42.485804 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 03:05:42.485874 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 03:05:42.487268 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:42.480626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34319 (local address 127.6.192.3:53575)
0928 03:05:42.481554 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:42.481591 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:42.483330 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:42.484145 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:42.484154 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:42.484254 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:42.485009 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:05:42.485011 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:42.485052 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:05:42.485144 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:42.485519 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:42.485547 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:42.485557 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 03:05:42.485864 (+ 307us) client_negotiation.cc:240] Negotiation successful
0928 03:05:42.486114 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":297,"threads_started":1}
I20260928 03:05:42.488682 17225 heartbeater.cc:347] Connected to a master server at 127.6.192.62:34319
I20260928 03:05:42.489027 17225 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:42.489647 17225 heartbeater.cc:511] Master 127.6.192.62:34319 requested a full tablet report, sending...
I20260928 03:05:42.491134 16776 ts_manager.cc:194] Registered new tserver with Master: c3b9ed00864a43969bc42c950840064f (127.6.192.3:38445)
I20260928 03:05:42.500270 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 03:05:42.506768 16965 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:05:42.505121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59422 (local address 127.6.192.62:34319)
0928 03:05:42.505289 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:42.505294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:42.505800 (+ 506us) server_negotiation.cc:407] Connection header received
0928 03:05:42.505973 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:42.505980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:42.506058 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:42.506169 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:42.506495 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59422: BlockingRecv error: recv got EOF from 127.0.0.1:59422 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 03:05:42.507843 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 16836
I20260928 03:05:42.511564 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:42.508889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.6.192.1:38223)
0928 03:05:42.509437 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:42.509444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:42.509470 (+ 26us) server_negotiation.cc:407] Connection header received
0928 03:05:42.509530 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:42.509536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:42.509751 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:42.509875 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:42.510872 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:42.510879 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:42.510891 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:05:42.510911 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:42.510954 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:05:42.510968 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:42.511083 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:42.511175 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 03:05:42.511256 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":269,"threads_started":1}
I20260928 03:05:42.512764 16958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:42.594234 16963 heartbeater.cc:503] Master 127.6.192.62:34319 was elected leader, sending a full tablet report...
I20260928 03:05:42.595847 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 16836
W20260928 03:05:42.609828 16764 connection.cc:535] server connection from 127.6.192.1:46949 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 03:05:42.610154 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 16967
I20260928 03:05:42.613711 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:42.611042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37872 (local address 127.6.192.2:45773)
0928 03:05:42.611600 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:42.611608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:42.611636 (+ 28us) server_negotiation.cc:407] Connection header received
0928 03:05:42.611699 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:42.611705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:42.611925 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:42.612071 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:42.613014 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:42.613021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:42.613033 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:05:42.613046 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:42.613093 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:05:42.613110 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:42.613220 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:42.613396 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 03:05:42.613490 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":263,"threads_started":1}
I20260928 03:05:42.615051 17090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:42.698016 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 16967
I20260928 03:05:42.715863 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 17098
I20260928 03:05:42.720603 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:42.716940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55624 (local address 127.6.192.3:38445)
0928 03:05:42.717672 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:42.717687 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:42.717721 (+ 34us) server_negotiation.cc:407] Connection header received
0928 03:05:42.717788 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:42.717795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:42.718138 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:42.718338 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:42.719769 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:42.719775 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:42.719790 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:05:42.719807 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:42.719850 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:05:42.719864 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:42.720068 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:42.720151 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 03:05:42.720263 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":356,"threads_started":1}
I20260928 03:05:42.721872 17220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:42.808885 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 17098
I20260928 03:05:42.825093 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 16746
I20260928 03:05:42.826426 16807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:42.932235 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 16746
2026-09-28T03:05:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2486 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:05:43 dist-test-slave-fjww krb5kdc[17243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:05:43 dist-test-slave-fjww krb5kdc[17243](info): set up 2 sockets
Sep 28 03:05:43 dist-test-slave-fjww krb5kdc[17243](info): commencing operation
krb5kdc: starting...
W20260928 03:05:44.109223 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:05:44 dist-test-slave-fjww krb5kdc[17243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:05:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:05:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:05:44.364336 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:34895
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:34895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:05:44.710711 17256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:44.711020 17256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:44.711206 17256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:44.718921 17256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:05:44.719130 17256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:44.719254 17256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:44.719374 17256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:05:44.719519 17256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:05:44.730042 17256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:34895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:34895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:44.731596 17256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:44.733071 17256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:44.746085 17261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:44.746105 17262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:44.746244 17264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:44.747123 17256 server_base.cc:1034] running on GCE node
I20260928 03:05:44.748073 17256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:44.749718 17256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:44.751010 17256 hybrid_clock.cc:648] HybridClock initialized: now 1790564744750933 us; error 72 us; skew 500 ppm
I20260928 03:05:44.751609 17256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:44 dist-test-slave-fjww krb5kdc[17243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564744, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:44.755296 17256 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:05:44.757174 17256 webserver.cc:460] Webserver started at http://127.6.192.62:39925/ using document root <none> and password file <none>
I20260928 03:05:44.757967 17256 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:44.758129 17256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:44.758507 17256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:44.762184 17256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "20b65989dd1a4fbf961c505d6f843a50"
format_stamp: "Formatted at 2026-09-28 03:05:44 on dist-test-slave-fjww"
server_key: "8c54f1283ca5b6e7b2055da32ad32927"
server_key_iv: "9034df7ce0987f708698fbed8d58e9c2"
server_key_version: "encryptionkey@0"
I20260928 03:05:44.763094 17256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "20b65989dd1a4fbf961c505d6f843a50"
format_stamp: "Formatted at 2026-09-28 03:05:44 on dist-test-slave-fjww"
server_key: "8c54f1283ca5b6e7b2055da32ad32927"
server_key_iv: "9034df7ce0987f708698fbed8d58e9c2"
server_key_version: "encryptionkey@0"
I20260928 03:05:44.769246 17256 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 03:05:44.773313 17271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:44.774972 17256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 03:05:44.775218 17256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "20b65989dd1a4fbf961c505d6f843a50"
format_stamp: "Formatted at 2026-09-28 03:05:44 on dist-test-slave-fjww"
server_key: "8c54f1283ca5b6e7b2055da32ad32927"
server_key_iv: "9034df7ce0987f708698fbed8d58e9c2"
server_key_version: "encryptionkey@0"
I20260928 03:05:44.775507 17256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:44.813544 17256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:44.818320 17256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:44.818638 17256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:44.839210 17256 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:34895
I20260928 03:05:44.839241 17322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:34895 every 8 connection(s)
I20260928 03:05:44.840823 17256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:05:44.845324 17323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:44.850648 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 17256
I20260928 03:05:44.850915 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:05:44.851683 6912 external_mini_cluster.cc:1428] Setting key a67edb02168f9ccd982f778900f9030d
I20260928 03:05:44.858088 17323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50: Bootstrap starting.
I20260928 03:05:44.861863 17323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:44.863525 17323 log.cc:824] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50: Log is configured to *not* fsync() on all Append() calls
Sep 28 03:05:44 dist-test-slave-fjww krb5kdc[17243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:05:44.867506 17323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50: No bootstrap required, opened a new log
I20260928 03:05:44.878655 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:44.853921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.6.192.62:34895)
0928 03:05:44.854840 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:44.854856 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:44.854904 (+ 48us) server_negotiation.cc:407] Connection header received
0928 03:05:44.856796 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:44.856824 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:44.857253 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:44.857717 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:44.859270 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:44.860760 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:44.862170 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:44.863480 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:44.863753 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:44.868588 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:44.868600 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:44.868621 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:44.868674 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:44.873119 (+ 4445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:44.873966 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:44.873976 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:44.873986 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:44.874224 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:44.874636 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:44.874642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:44.874646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:44.875688 (+ 1042us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:44.876067 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:44.876424 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 03:05:44.876977 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":419,"threads_started":1}
I20260928 03:05:44.880077 17323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20b65989dd1a4fbf961c505d6f843a50" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34895 } }
I20260928 03:05:44.880642 17323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:44.880793 17323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b65989dd1a4fbf961c505d6f843a50, State: Initialized, Role: FOLLOWER
I20260928 03:05:44.881500 17323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20b65989dd1a4fbf961c505d6f843a50" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34895 } }
I20260928 03:05:44.881803 17323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:05:44.881985 17323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:05:44.882226 17323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:44.885061 17323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20b65989dd1a4fbf961c505d6f843a50" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34895 } }
I20260928 03:05:44.885694 17323 leader_election.cc:302] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20b65989dd1a4fbf961c505d6f843a50; no voters:
I20260928 03:05:44.886487 17323 leader_election.cc:288] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:05:44.886600 17328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:05:44.887555 17328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50 [term 1 LEADER]: Becoming Leader. State: Replica: 20b65989dd1a4fbf961c505d6f843a50, State: Running, Role: LEADER
I20260928 03:05:44.888144 17328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20b65989dd1a4fbf961c505d6f843a50" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34895 } }
I20260928 03:05:44.889118 17323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:05:44.891177 17329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20b65989dd1a4fbf961c505d6f843a50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20b65989dd1a4fbf961c505d6f843a50" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34895 } } }
I20260928 03:05:44.891708 17329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50 [sys.catalog]: This master's current role is: LEADER
I20260928 03:05:44.892385 17330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20b65989dd1a4fbf961c505d6f843a50. Latest consensus state: current_term: 1 leader_uuid: "20b65989dd1a4fbf961c505d6f843a50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20b65989dd1a4fbf961c505d6f843a50" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34895 } } }
I20260928 03:05:44.892752 17336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:05:44.892846 17330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50 [sys.catalog]: This master's current role is: LEADER
I20260928 03:05:44.899200 17336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:05:44.911060 17336 catalog_manager.cc:1347] Generated new cluster ID: df8a6cac09a24bbe82d808f1f5304ac9
I20260928 03:05:44.911224 17336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:05:44.930750 17336 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:05:44.932380 17336 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:05:44.941102 17336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50: Generated new TSK 0
I20260928 03:05:44.942021 17336 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
I20260928 03:05:45.016553 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34895
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:05:45.381721 17349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:45.382100 17349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:45.382292 17349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:45.390208 17349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:45.390414 17349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:45.390635 17349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:05:45.401348 17349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:45.402822 17349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:45.404320 17349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:45.418552 17354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:45.418862 17355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:45.419282 17357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:45.420799 17349 server_base.cc:1034] running on GCE node
I20260928 03:05:45.421471 17349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:45.422499 17349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:45.423666 17349 hybrid_clock.cc:648] HybridClock initialized: now 1790564745423592 us; error 69 us; skew 500 ppm
I20260928 03:05:45.424039 17349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:45 dist-test-slave-fjww krb5kdc[17243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564745, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:45.427735 17349 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:05:45.429436 17349 webserver.cc:460] Webserver started at http://127.6.192.1:35535/ using document root <none> and password file <none>
I20260928 03:05:45.430197 17349 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:45.430342 17349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:45.430665 17349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:45.433892 17349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "cd748e71113842b0b339eb42c7e9da95"
format_stamp: "Formatted at 2026-09-28 03:05:45 on dist-test-slave-fjww"
server_key: "efd52a494e1386aa79ed5dda97dc6caf"
server_key_iv: "2a8d865a54f643732f4076a954d4c621"
server_key_version: "encryptionkey@0"
I20260928 03:05:45.434741 17349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "cd748e71113842b0b339eb42c7e9da95"
format_stamp: "Formatted at 2026-09-28 03:05:45 on dist-test-slave-fjww"
server_key: "efd52a494e1386aa79ed5dda97dc6caf"
server_key_iv: "2a8d865a54f643732f4076a954d4c621"
server_key_version: "encryptionkey@0"
I20260928 03:05:45.440442 17349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 03:05:45.444361 17364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:45.445936 17349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:05:45.446141 17349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "cd748e71113842b0b339eb42c7e9da95"
format_stamp: "Formatted at 2026-09-28 03:05:45 on dist-test-slave-fjww"
server_key: "efd52a494e1386aa79ed5dda97dc6caf"
server_key_iv: "2a8d865a54f643732f4076a954d4c621"
server_key_version: "encryptionkey@0"
I20260928 03:05:45.446354 17349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:45.466234 17349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:45.470916 17349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:45.471261 17349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:45.472790 17349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:05:45.474514 17349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:45.474599 17349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:45.474759 17349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:45.474825 17349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:45.516628 17349 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:34819
I20260928 03:05:45.516741 17476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:34819 every 8 connection(s)
I20260928 03:05:45.518103 17349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:05:45.525844 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 17349
I20260928 03:05:45.526114 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:05:45.526868 6912 external_mini_cluster.cc:1428] Setting key c5ff00636439ac8053c777f0bdf64685
Sep 28 03:05:45 dist-test-slave-fjww krb5kdc[17243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564745, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:05:45.542562 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:45.521132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:51951 (local address 127.6.192.62:34895)
0928 03:05:45.521676 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:45.521684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:45.522805 (+ 1121us) server_negotiation.cc:407] Connection header received
0928 03:05:45.524737 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:45.524742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:45.524814 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:45.524925 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:45.526786 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:45.527926 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:45.530146 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:45.530971 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:45.531165 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:45.535339 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:45.535349 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:45.535357 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:45.535414 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:45.538689 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:45.539821 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:45.539826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:45.539829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:45.539979 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:45.540541 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:45.540546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:45.540548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:45.540856 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:45.541023 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:45.542040 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0928 03:05:45.542258 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":254,"threads_started":1}
I20260928 03:05:45.544075 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:45.521696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34895 (local address 127.6.192.1:51951)
0928 03:05:45.522612 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:45.522656 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:45.524438 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:45.525197 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:45.525209 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:45.525870 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:45.526580 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:45.526599 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:45.528167 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:45.528173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:45.529959 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:45.529972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:45.531273 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:45.531280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:45.531725 (+ 445us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:45.532518 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:45.532545 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:45.535039 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:45.538875 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:45.538889 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:45.538921 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:45.539661 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:45.540102 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:45.540109 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:45.540115 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:45.540377 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:45.541037 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:45.541071 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:45.541560 (+ 489us) client_negotiation.cc:769] Sending connection context
0928 03:05:45.541870 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 03:05:45.542496 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":338,"threads_started":1}
I20260928 03:05:45.545997 17477 heartbeater.cc:347] Connected to a master server at 127.6.192.62:34895
I20260928 03:05:45.546347 17477 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:45.547222 17477 heartbeater.cc:511] Master 127.6.192.62:34895 requested a full tablet report, sending...
I20260928 03:05:45.549948 17288 ts_manager.cc:194] Registered new tserver with Master: cd748e71113842b0b339eb42c7e9da95 (127.6.192.1:34819)
WARNING: no policy specified for kudu/127.6.192.2@KRBTEST.COM; defaulting to no policy
I20260928 03:05:45.552187 17288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:51951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
I20260928 03:05:45.587984 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34895
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:05:45.939548 17483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:45.939931 17483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:45.940132 17483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:45.949009 17483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:45.949186 17483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:45.949373 17483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:05:45.960996 17483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:45.962584 17483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:45.964116 17483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:45.978334 17491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:45.978294 17489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:45.978372 17488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:45.979540 17483 server_base.cc:1034] running on GCE node
I20260928 03:05:45.980185 17483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:45.981289 17483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:45.982604 17483 hybrid_clock.cc:648] HybridClock initialized: now 1790564745982552 us; error 36 us; skew 500 ppm
I20260928 03:05:45.982970 17483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:45 dist-test-slave-fjww krb5kdc[17243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564745, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:45.986662 17483 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:05:45.988322 17483 webserver.cc:460] Webserver started at http://127.6.192.2:36723/ using document root <none> and password file <none>
I20260928 03:05:45.989166 17483 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:45.989310 17483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:45.989650 17483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:45.993117 17483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "441b104b3e3b405e835601a8682c25b0"
format_stamp: "Formatted at 2026-09-28 03:05:45 on dist-test-slave-fjww"
server_key: "ad48a6fe9771ec351a2e9087f71e5f2c"
server_key_iv: "73ae5cfb5bebc1df06e4a0f95b4f9dd0"
server_key_version: "encryptionkey@0"
I20260928 03:05:45.994005 17483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "441b104b3e3b405e835601a8682c25b0"
format_stamp: "Formatted at 2026-09-28 03:05:45 on dist-test-slave-fjww"
server_key: "ad48a6fe9771ec351a2e9087f71e5f2c"
server_key_iv: "73ae5cfb5bebc1df06e4a0f95b4f9dd0"
server_key_version: "encryptionkey@0"
I20260928 03:05:46.000530 17483 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 03:05:46.004629 17498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:46.006402 17483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 03:05:46.006708 17483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "441b104b3e3b405e835601a8682c25b0"
format_stamp: "Formatted at 2026-09-28 03:05:45 on dist-test-slave-fjww"
server_key: "ad48a6fe9771ec351a2e9087f71e5f2c"
server_key_iv: "73ae5cfb5bebc1df06e4a0f95b4f9dd0"
server_key_version: "encryptionkey@0"
I20260928 03:05:46.006935 17483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:46.035308 17483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:46.040087 17483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:46.040426 17483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:46.041512 17483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:05:46.043079 17483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:46.043155 17483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:46.043306 17483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:46.043359 17483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:46.080463 17483 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:37695
I20260928 03:05:46.080507 17610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:37695 every 8 connection(s)
I20260928 03:05:46.081944 17483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:05:46.083616 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 17483
I20260928 03:05:46.083794 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:05:46.084187 6912 external_mini_cluster.cc:1428] Setting key 87628cd4bd5bc61f3004baaddd347506
Sep 28 03:05:46 dist-test-slave-fjww krb5kdc[17243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564745, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:05:46.108165 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:46.086365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:42353 (local address 127.6.192.62:34895)
0928 03:05:46.086904 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:46.086910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:46.088170 (+ 1260us) server_negotiation.cc:407] Connection header received
0928 03:05:46.090162 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:46.090171 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:46.090268 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:46.090388 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:46.092096 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.093333 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.095164 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.095747 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.095903 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:46.100548 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:46.100554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:46.100558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:46.100595 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:46.103798 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:46.105071 (+ 1273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:46.105076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:46.105078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:46.105287 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:46.106022 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:46.106027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:46.106030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:46.106416 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:46.106739 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:46.107684 (+ 945us) server_negotiation.cc:299] Negotiation successful
0928 03:05:46.107872 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":302,"threads_started":1}
WARNING: no policy specified for kudu/127.6.192.3@KRBTEST.COM; defaulting to no policy
I20260928 03:05:46.109663 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:46.087055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34895 (local address 127.6.192.2:42353)
0928 03:05:46.087987 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:46.088042 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:46.089843 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:46.090636 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:46.090644 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:46.091212 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:46.091920 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:46.091939 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.093547 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.093552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:46.094974 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:46.094988 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.096017 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.096027 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:46.096289 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:46.097105 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:46.097130 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:46.100155 (+ 3025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:46.103994 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:46.104010 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:46.104034 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:46.104881 (+ 847us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:46.105452 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:46.105460 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:46.105467 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:46.105808 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:46.106587 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:46.106598 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:46.107110 (+ 512us) client_negotiation.cc:769] Sending connection context
0928 03:05:46.107522 (+ 412us) client_negotiation.cc:240] Negotiation successful
0928 03:05:46.108181 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":443,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.3@KRBTEST.COM" created.
I20260928 03:05:46.111650 17611 heartbeater.cc:347] Connected to a master server at 127.6.192.62:34895
I20260928 03:05:46.112119 17611 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:46.113101 17611 heartbeater.cc:511] Master 127.6.192.62:34895 requested a full tablet report, sending...
I20260928 03:05:46.114871 17288 ts_manager.cc:194] Registered new tserver with Master: 441b104b3e3b405e835601a8682c25b0 (127.6.192.2:37695)
I20260928 03:05:46.115800 17288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:42353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
I20260928 03:05:46.144555 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34895
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:05:46.494060 17617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:05:46.494450 17617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:05:46.494645 17617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:05:46.502388 17617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:05:46.502578 17617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:05:46.502781 17617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:05:46.513929 17617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:05:46.515369 17617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:05:46.517338 17617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:05:46.533059 17622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:46.533433 17623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:05:46.534055 17625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:05:46.534582 17617 server_base.cc:1034] running on GCE node
I20260928 03:05:46.535341 17617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:05:46.536509 17617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:05:46.537724 17617 hybrid_clock.cc:648] HybridClock initialized: now 1790564746537675 us; error 41 us; skew 500 ppm
I20260928 03:05:46.538074 17617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:05:46 dist-test-slave-fjww krb5kdc[17243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564746, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:05:46.541594 17617 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:05:46.543164 17617 webserver.cc:460] Webserver started at http://127.6.192.3:36139/ using document root <none> and password file <none>
I20260928 03:05:46.543977 17617 fs_manager.cc:359] Metadata directory not provided
I20260928 03:05:46.544134 17617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:05:46.544458 17617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:05:46.547670 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "97b8742b68e8447880e9be328bfc9a3c"
format_stamp: "Formatted at 2026-09-28 03:05:46 on dist-test-slave-fjww"
server_key: "1d331a78bde0e41755c6bef26d453735"
server_key_iv: "e33860e5ad3290fa7d6c510f62cd521c"
server_key_version: "encryptionkey@0"
I20260928 03:05:46.548493 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "97b8742b68e8447880e9be328bfc9a3c"
format_stamp: "Formatted at 2026-09-28 03:05:46 on dist-test-slave-fjww"
server_key: "1d331a78bde0e41755c6bef26d453735"
server_key_iv: "e33860e5ad3290fa7d6c510f62cd521c"
server_key_version: "encryptionkey@0"
I20260928 03:05:46.554149 17617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 03:05:46.556137 17477 heartbeater.cc:503] Master 127.6.192.62:34895 was elected leader, sending a full tablet report...
I20260928 03:05:46.558467 17632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:46.560068 17617 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 03:05:46.560272 17617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "97b8742b68e8447880e9be328bfc9a3c"
format_stamp: "Formatted at 2026-09-28 03:05:46 on dist-test-slave-fjww"
server_key: "1d331a78bde0e41755c6bef26d453735"
server_key_iv: "e33860e5ad3290fa7d6c510f62cd521c"
server_key_version: "encryptionkey@0"
I20260928 03:05:46.560501 17617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:05:46.574026 17617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:05:46.578475 17617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:05:46.578838 17617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:05:46.579995 17617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:05:46.581463 17617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:05:46.581573 17617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:46.581743 17617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:05:46.581815 17617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:05:46.616685 17617 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:41387
I20260928 03:05:46.616735 17745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:41387 every 8 connection(s)
I20260928 03:05:46.618176 17617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:05:46.619848 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 17617
I20260928 03:05:46.620054 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:05:46.620669 6912 external_mini_cluster.cc:1428] Setting key 3719305297cace3d7fec94d8476f1d1f
Sep 28 03:05:46 dist-test-slave-fjww krb5kdc[17243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564746, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:05:46.640125 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:46.621319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:47973 (local address 127.6.192.62:34895)
0928 03:05:46.621835 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:46.621839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:46.622993 (+ 1154us) server_negotiation.cc:407] Connection header received
0928 03:05:46.625232 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:46.625236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:46.625329 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:46.625422 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:46.626923 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.627834 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.629493 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.629887 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.629987 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:46.633741 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:46.633746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:46.633749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:46.633785 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:46.637102 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:46.637883 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:46.637887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:46.637889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:46.638026 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:46.638428 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:46.638431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:46.638440 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:46.638708 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:46.638825 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:46.639638 (+ 813us) server_negotiation.cc:299] Negotiation successful
0928 03:05:46.639832 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":231,"threads_started":1}
I20260928 03:05:46.641141 17749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:46.621956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34895 (local address 127.6.192.3:47973)
0928 03:05:46.622804 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:46.622865 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:46.624953 (+ 2088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:46.625618 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:46.625627 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:46.626158 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:05:46.626758 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:46.626776 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.628038 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.628046 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:46.629342 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:46.629355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.630111 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.630117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:46.630328 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:46.631067 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:05:46.631090 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:05:46.633515 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:05:46.637235 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:46.637243 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:46.637258 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:46.637764 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:46.638129 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:05:46.638132 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:05:46.638134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:05:46.638328 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:05:46.638817 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:05:46.638824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:05:46.639164 (+ 340us) client_negotiation.cc:769] Sending connection context
0928 03:05:46.639410 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 03:05:46.639909 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":320,"threads_started":1}
I20260928 03:05:46.642711 17746 heartbeater.cc:347] Connected to a master server at 127.6.192.62:34895
I20260928 03:05:46.643041 17746 heartbeater.cc:464] Registering TS with master...
I20260928 03:05:46.643882 17746 heartbeater.cc:511] Master 127.6.192.62:34895 requested a full tablet report, sending...
I20260928 03:05:46.645421 17288 ts_manager.cc:194] Registered new tserver with Master: 97b8742b68e8447880e9be328bfc9a3c (127.6.192.3:41387)
I20260928 03:05:46.646265 17288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:47973
I20260928 03:05:46.649564 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:05:46.668015 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:46.654384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57692 (local address 127.6.192.62:34895)
0928 03:05:46.654781 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:46.654789 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:46.655144 (+ 355us) server_negotiation.cc:407] Connection header received
0928 03:05:46.655361 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:46.655365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:46.655577 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:46.655690 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:46.657124 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.657871 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.659103 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.659579 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.659723 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:46.662122 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:46.662127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:46.662130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:46.662164 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:46.665084 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:46.665869 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:46.665880 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:46.665882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:46.666010 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:46.666412 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:46.666417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:46.666421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:46.666774 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:46.667035 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:46.667421 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 03:05:46.667692 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260928 03:05:46.673004 17288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:05:46.680819 17288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:05:46.713173 17760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:46.694448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:34819 (local address 127.0.0.1:41412)
0928 03:05:46.701767 (+ 7319us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:46.701822 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:46.702095 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:46.704457 (+ 2362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:46.704467 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:46.704529 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:46.705639 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:46.705654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.707405 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.707413 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:46.710079 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:46.710105 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.711869 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.711876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:46.712104 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:46.712178 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 03:05:46.712329 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 03:05:46.712586 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7007,"mutex_wait_us":63,"thread_start_us":7838,"threads_started":1}
I20260928 03:05:46.713506 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:46.694725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:37695 (local address 127.0.0.1:58530)
0928 03:05:46.702668 (+ 7943us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:46.702694 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:46.702843 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:46.704639 (+ 1796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:46.704645 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:46.704679 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:46.705278 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:46.705293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.707880 (+ 2587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.707889 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:46.710119 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:46.710128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.712403 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.712409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:46.712602 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:46.712638 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 03:05:46.712713 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 03:05:46.712867 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7729,"mutex_wait_us":7,"thread_start_us":6540,"threads_started":1}
I20260928 03:05:46.714439 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:46.694825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41412 (local address 127.6.192.1:34819)
0928 03:05:46.703653 (+ 8828us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:46.703663 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:46.703697 (+ 34us) server_negotiation.cc:407] Connection header received
0928 03:05:46.703820 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:46.703832 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:46.704157 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:46.705086 (+ 929us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:46.705838 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.707269 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.710321 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.711641 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.711782 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:46.712046 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:46.713514 (+ 1468us) server_negotiation.cc:299] Negotiation successful
0928 03:05:46.714073 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8561,"thread_start_us":8350,"threads_started":1}
I20260928 03:05:46.714671 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:46.704285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:41387 (local address 127.0.0.1:57208)
0928 03:05:46.705023 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:46.705049 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:46.705267 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:46.706162 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:46.706166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:46.706195 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:46.706562 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:46.706573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.708682 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.708688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:46.711353 (+ 2665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:46.711371 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.713769 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.713778 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:46.713951 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:46.713987 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 03:05:46.714121 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 03:05:46.714264 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":1,"thread_start_us":329,"threads_started":1}
I20260928 03:05:46.714754 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:46.703246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57208 (local address 127.6.192.3:41387)
0928 03:05:46.704347 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:46.704358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:46.705145 (+ 787us) server_negotiation.cc:407] Connection header received
0928 03:05:46.705537 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:46.705545 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:46.705789 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:46.706012 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:46.706822 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.708490 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.711586 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.713202 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.713343 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:46.713588 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:46.714231 (+ 643us) server_negotiation.cc:299] Negotiation successful
0928 03:05:46.714416 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":893,"thread_start_us":690,"threads_started":1}
I20260928 03:05:46.715298 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:46.695302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58530 (local address 127.6.192.2:37695)
0928 03:05:46.703939 (+ 8637us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:46.703952 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:46.703987 (+ 35us) server_negotiation.cc:407] Connection header received
0928 03:05:46.704087 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:46.704094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:46.704340 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:46.705551 (+ 1211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:46.705906 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.707732 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.710292 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.712123 (+ 1831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.713904 (+ 1781us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:46.714160 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:46.714246 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 03:05:46.714434 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8412,"thread_start_us":8259,"threads_started":1}
I20260928 03:05:46.718652 17546 tablet_service.cc:1467] Processing CreateTablet for tablet e0f8960231a245ae9d0007ddb95587f9 (DEFAULT_TABLE table=test-table [id=43526846cf7e48e5bcd409428b889dc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:46.718672 17680 tablet_service.cc:1467] Processing CreateTablet for tablet e0f8960231a245ae9d0007ddb95587f9 (DEFAULT_TABLE table=test-table [id=43526846cf7e48e5bcd409428b889dc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:46.718672 17412 tablet_service.cc:1467] Processing CreateTablet for tablet e0f8960231a245ae9d0007ddb95587f9 (DEFAULT_TABLE table=test-table [id=43526846cf7e48e5bcd409428b889dc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:05:46.720180 17680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f8960231a245ae9d0007ddb95587f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:46.720160 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f8960231a245ae9d0007ddb95587f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:46.720616 17412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f8960231a245ae9d0007ddb95587f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:05:46.734519 17765 tablet_bootstrap.cc:492] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0: Bootstrap starting.
I20260928 03:05:46.736925 17766 tablet_bootstrap.cc:492] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95: Bootstrap starting.
I20260928 03:05:46.738476 17767 tablet_bootstrap.cc:492] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c: Bootstrap starting.
I20260928 03:05:46.739368 17765 tablet_bootstrap.cc:654] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:46.740916 17765 log.cc:824] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0: Log is configured to *not* fsync() on all Append() calls
I20260928 03:05:46.742690 17767 tablet_bootstrap.cc:654] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:46.744200 17767 log.cc:824] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c: Log is configured to *not* fsync() on all Append() calls
I20260928 03:05:46.744484 17765 tablet_bootstrap.cc:492] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0: No bootstrap required, opened a new log
I20260928 03:05:46.744786 17765 ts_tablet_manager.cc:1397] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20260928 03:05:46.745431 17766 tablet_bootstrap.cc:654] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95: Neither blocks nor log segments found. Creating new log.
I20260928 03:05:46.746797 17766 log.cc:824] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95: Log is configured to *not* fsync() on all Append() calls
I20260928 03:05:46.747822 17767 tablet_bootstrap.cc:492] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c: No bootstrap required, opened a new log
I20260928 03:05:46.748179 17767 ts_tablet_manager.cc:1397] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 03:05:46.752362 17766 tablet_bootstrap.cc:492] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95: No bootstrap required, opened a new log
I20260928 03:05:46.752681 17766 ts_tablet_manager.cc:1397] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.008s
I20260928 03:05:46.753873 17765 raft_consensus.cc:348] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd748e71113842b0b339eb42c7e9da95" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34819 } } peers { permanent_uuid: "441b104b3e3b405e835601a8682c25b0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37695 } } peers { permanent_uuid: "97b8742b68e8447880e9be328bfc9a3c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41387 } }
I20260928 03:05:46.754279 17765 raft_consensus.cc:374] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:46.754379 17765 raft_consensus.cc:729] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441b104b3e3b405e835601a8682c25b0, State: Initialized, Role: FOLLOWER
I20260928 03:05:46.755025 17765 consensus_queue.cc:260] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd748e71113842b0b339eb42c7e9da95" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34819 } } peers { permanent_uuid: "441b104b3e3b405e835601a8682c25b0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37695 } } peers { permanent_uuid: "97b8742b68e8447880e9be328bfc9a3c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41387 } }
I20260928 03:05:46.756237 17611 heartbeater.cc:503] Master 127.6.192.62:34895 was elected leader, sending a full tablet report...
I20260928 03:05:46.756791 17765 ts_tablet_manager.cc:1428] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0: Time spent starting tablet: real 0.012s user 0.007s sys 0.006s
I20260928 03:05:46.762219 17767 raft_consensus.cc:348] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd748e71113842b0b339eb42c7e9da95" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34819 } } peers { permanent_uuid: "441b104b3e3b405e835601a8682c25b0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37695 } } peers { permanent_uuid: "97b8742b68e8447880e9be328bfc9a3c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41387 } }
I20260928 03:05:46.762810 17767 raft_consensus.cc:374] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:46.762923 17767 raft_consensus.cc:729] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97b8742b68e8447880e9be328bfc9a3c, State: Initialized, Role: FOLLOWER
I20260928 03:05:46.763756 17767 consensus_queue.cc:260] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd748e71113842b0b339eb42c7e9da95" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34819 } } peers { permanent_uuid: "441b104b3e3b405e835601a8682c25b0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37695 } } peers { permanent_uuid: "97b8742b68e8447880e9be328bfc9a3c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41387 } }
I20260928 03:05:46.764709 17766 raft_consensus.cc:348] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd748e71113842b0b339eb42c7e9da95" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34819 } } peers { permanent_uuid: "441b104b3e3b405e835601a8682c25b0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37695 } } peers { permanent_uuid: "97b8742b68e8447880e9be328bfc9a3c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41387 } }
I20260928 03:05:46.765110 17746 heartbeater.cc:503] Master 127.6.192.62:34895 was elected leader, sending a full tablet report...
I20260928 03:05:46.765249 17766 raft_consensus.cc:374] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:05:46.765380 17766 raft_consensus.cc:729] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd748e71113842b0b339eb42c7e9da95, State: Initialized, Role: FOLLOWER
I20260928 03:05:46.765913 17767 ts_tablet_manager.cc:1428] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c: Time spent starting tablet: real 0.018s user 0.007s sys 0.008s
I20260928 03:05:46.766110 17766 consensus_queue.cc:260] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd748e71113842b0b339eb42c7e9da95" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34819 } } peers { permanent_uuid: "441b104b3e3b405e835601a8682c25b0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37695 } } peers { permanent_uuid: "97b8742b68e8447880e9be328bfc9a3c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41387 } }
I20260928 03:05:46.768067 17766 ts_tablet_manager.cc:1428] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
W20260928 03:05:46.770828 17478 tablet.cc:2369] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:05:46.834228 17612 tablet.cc:2369] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:05:46.857729 17771 raft_consensus.cc:482] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:05:46.858064 17771 raft_consensus.cc:504] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd748e71113842b0b339eb42c7e9da95" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34819 } } peers { permanent_uuid: "441b104b3e3b405e835601a8682c25b0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37695 } } peers { permanent_uuid: "97b8742b68e8447880e9be328bfc9a3c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41387 } }
I20260928 03:05:46.859848 17771 leader_election.cc:288] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd748e71113842b0b339eb42c7e9da95 (127.6.192.1:34819), 97b8742b68e8447880e9be328bfc9a3c (127.6.192.3:41387)
I20260928 03:05:46.868140 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:46.860291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:54557 (local address 127.6.192.1:34819)
0928 03:05:46.860489 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:46.860494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:46.861142 (+ 648us) server_negotiation.cc:407] Connection header received
0928 03:05:46.861380 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:46.861388 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:46.861503 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:46.861620 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:46.862286 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.863517 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.865969 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.867053 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.867172 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:46.867277 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:46.867732 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 03:05:46.867885 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 03:05:46.868184 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:46.860298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:34819 (local address 127.6.192.2:54557)
0928 03:05:46.860972 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:46.860989 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:46.861185 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:46.861660 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:46.861665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:46.861692 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:46.862134 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:46.862150 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.863710 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.863717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:46.865780 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:46.865796 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.867304 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.867310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:46.867531 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:46.867568 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 03:05:46.867669 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 03:05:46.867801 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":28,"thread_start_us":403,"threads_started":1}
I20260928 03:05:46.868588 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:46.861278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:49197 (local address 127.6.192.3:41387)
0928 03:05:46.861524 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:46.861530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:46.861940 (+ 410us) server_negotiation.cc:407] Connection header received
0928 03:05:46.862101 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:46.862105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:46.862169 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:46.862287 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:05:46.862770 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.863971 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.866448 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.867371 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.867545 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:46.867659 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:46.868191 (+ 532us) server_negotiation.cc:299] Negotiation successful
0928 03:05:46.868344 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 03:05:46.869050 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:46.861107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:41387 (local address 127.6.192.2:49197)
0928 03:05:46.861820 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0928 03:05:46.861844 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 03:05:46.861967 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:05:46.862304 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:05:46.862307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:05:46.862326 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:05:46.862602 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:46.862609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.864148 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.864153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:46.866260 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:05:46.866277 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.867769 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.867775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:05:46.867977 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:46.868015 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 03:05:46.868119 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 03:05:46.868275 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":2,"thread_start_us":361,"threads_started":1}
W20260928 03:05:46.869951 17747 tablet.cc:2369] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:05:46.869863 17432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f8960231a245ae9d0007ddb95587f9" candidate_uuid: "441b104b3e3b405e835601a8682c25b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd748e71113842b0b339eb42c7e9da95" is_pre_election: true
I20260928 03:05:46.870157 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f8960231a245ae9d0007ddb95587f9" candidate_uuid: "441b104b3e3b405e835601a8682c25b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97b8742b68e8447880e9be328bfc9a3c" is_pre_election: true
I20260928 03:05:46.870467 17432 raft_consensus.cc:2454] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 441b104b3e3b405e835601a8682c25b0 in term 0.
I20260928 03:05:46.870700 17700 raft_consensus.cc:2454] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 441b104b3e3b405e835601a8682c25b0 in term 0.
I20260928 03:05:46.871266 17502 leader_election.cc:302] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441b104b3e3b405e835601a8682c25b0, cd748e71113842b0b339eb42c7e9da95; no voters:
I20260928 03:05:46.871712 17771 raft_consensus.cc:2781] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:05:46.871874 17771 raft_consensus.cc:482] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:05:46.871990 17771 raft_consensus.cc:3037] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:46.874456 17771 raft_consensus.cc:504] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd748e71113842b0b339eb42c7e9da95" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34819 } } peers { permanent_uuid: "441b104b3e3b405e835601a8682c25b0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37695 } } peers { permanent_uuid: "97b8742b68e8447880e9be328bfc9a3c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41387 } }
I20260928 03:05:46.875157 17771 leader_election.cc:288] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [CANDIDATE]: Term 1 election: Requested vote from peers cd748e71113842b0b339eb42c7e9da95 (127.6.192.1:34819), 97b8742b68e8447880e9be328bfc9a3c (127.6.192.3:41387)
I20260928 03:05:46.875751 17432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f8960231a245ae9d0007ddb95587f9" candidate_uuid: "441b104b3e3b405e835601a8682c25b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd748e71113842b0b339eb42c7e9da95"
I20260928 03:05:46.875850 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f8960231a245ae9d0007ddb95587f9" candidate_uuid: "441b104b3e3b405e835601a8682c25b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97b8742b68e8447880e9be328bfc9a3c"
I20260928 03:05:46.875985 17432 raft_consensus.cc:3037] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:46.876165 17700 raft_consensus.cc:3037] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:05:46.878404 17700 raft_consensus.cc:2454] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441b104b3e3b405e835601a8682c25b0 in term 1.
I20260928 03:05:46.878409 17432 raft_consensus.cc:2454] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441b104b3e3b405e835601a8682c25b0 in term 1.
I20260928 03:05:46.878870 17502 leader_election.cc:302] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441b104b3e3b405e835601a8682c25b0, cd748e71113842b0b339eb42c7e9da95; no voters:
I20260928 03:05:46.879217 17771 raft_consensus.cc:2781] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:05:46.879773 17771 raft_consensus.cc:686] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [term 1 LEADER]: Becoming Leader. State: Replica: 441b104b3e3b405e835601a8682c25b0, State: Running, Role: LEADER
I20260928 03:05:46.880285 17771 consensus_queue.cc:237] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd748e71113842b0b339eb42c7e9da95" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34819 } } peers { permanent_uuid: "441b104b3e3b405e835601a8682c25b0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37695 } } peers { permanent_uuid: "97b8742b68e8447880e9be328bfc9a3c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41387 } }
I20260928 03:05:46.885537 17287 catalog_manager.cc:5495] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 441b104b3e3b405e835601a8682c25b0 (127.6.192.2). New cstate: current_term: 1 leader_uuid: "441b104b3e3b405e835601a8682c25b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd748e71113842b0b339eb42c7e9da95" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441b104b3e3b405e835601a8682c25b0" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 37695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97b8742b68e8447880e9be328bfc9a3c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41387 } health_report { overall_health: UNKNOWN } } }
I20260928 03:05:46.915316 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:46.909662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58544 (local address 127.6.192.2:37695)
0928 03:05:46.909947 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:46.909957 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:46.910050 (+ 93us) server_negotiation.cc:407] Connection header received
0928 03:05:46.910321 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:46.910325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:46.910404 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:46.910567 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:05:46.911006 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.911860 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.913205 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.913611 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.913747 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:46.914216 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:05:46.914725 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:05:46.914888 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:46.914946 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 03:05:46.915028 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260928 03:05:46.924000 17700 raft_consensus.cc:1261] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c [term 1 FOLLOWER]: Refusing update from remote peer 441b104b3e3b405e835601a8682c25b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:05:46.924085 17432 raft_consensus.cc:1261] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95 [term 1 FOLLOWER]: Refusing update from remote peer 441b104b3e3b405e835601a8682c25b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:05:46.925133 17771 consensus_queue.cc:1035] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97b8742b68e8447880e9be328bfc9a3c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:05:46.925518 17776 consensus_queue.cc:1035] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd748e71113842b0b339eb42c7e9da95" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:05:46.939229 17780 mvcc.cc:205] Tried to move back new op lower bound from 7334153203389472768 to 7334153203226722304. Current Snapshot: MvccSnapshot[applied={T|T < 7334153203389472768 or (T in {7334153203389472768})}]
I20260928 03:05:46.952842 17288 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:05:46.953145 17288 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:05:46.957006 17288 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20b65989dd1a4fbf961c505d6f843a50: Sending DeleteTablet for 3 replicas of tablet e0f8960231a245ae9d0007ddb95587f9
I20260928 03:05:46.958375 17546 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f8960231a245ae9d0007ddb95587f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:46 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:58530
I20260928 03:05:46.958362 17680 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f8960231a245ae9d0007ddb95587f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:46 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:57208
I20260928 03:05:46.959167 17412 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f8960231a245ae9d0007ddb95587f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:05:46 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:41412
I20260928 03:05:46.960877 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 17349
I20260928 03:05:46.971321 17789 tablet_replica.cc:330] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0: stopping tablet replica
I20260928 03:05:46.972112 17790 tablet_replica.cc:330] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c: stopping tablet replica
I20260928 03:05:46.972416 17791 tablet_replica.cc:330] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95: stopping tablet replica
I20260928 03:05:46.972656 17790 raft_consensus.cc:2229] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:05:46.972856 17791 raft_consensus.cc:2229] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:05:46.973286 17790 raft_consensus.cc:2258] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:46.973337 17791 raft_consensus.cc:2258] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:46.975562 17790 ts_tablet_manager.cc:1905] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:46.976481 17791 ts_tablet_manager.cc:1905] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:46.974105 17789 raft_consensus.cc:2229] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:05:46.979221 17789 raft_consensus.cc:2258] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:05:46.981876 17789 ts_tablet_manager.cc:1905] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:05:46.983311 17790 ts_tablet_manager.cc:1918] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:05:46.983575 17790 log.cc:1196] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/e0f8960231a245ae9d0007ddb95587f9
I20260928 03:05:46.984197 17790 ts_tablet_manager.cc:1939] T e0f8960231a245ae9d0007ddb95587f9 P 97b8742b68e8447880e9be328bfc9a3c: Deleting consensus metadata
I20260928 03:05:46.986255 17791 ts_tablet_manager.cc:1918] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:05:46.986243 17275 catalog_manager.cc:4841] TS 97b8742b68e8447880e9be328bfc9a3c (127.6.192.3:41387): tablet e0f8960231a245ae9d0007ddb95587f9 (table test-table [id=43526846cf7e48e5bcd409428b889dc3]) successfully deleted
I20260928 03:05:46.986438 17791 log.cc:1196] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/e0f8960231a245ae9d0007ddb95587f9
I20260928 03:05:46.987028 17791 ts_tablet_manager.cc:1939] T e0f8960231a245ae9d0007ddb95587f9 P cd748e71113842b0b339eb42c7e9da95: Deleting consensus metadata
I20260928 03:05:46.989102 17275 catalog_manager.cc:4841] TS cd748e71113842b0b339eb42c7e9da95 (127.6.192.1:34819): tablet e0f8960231a245ae9d0007ddb95587f9 (table test-table [id=43526846cf7e48e5bcd409428b889dc3]) successfully deleted
I20260928 03:05:46.989521 17789 ts_tablet_manager.cc:1918] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:05:46.989717 17789 log.cc:1196] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/e0f8960231a245ae9d0007ddb95587f9
I20260928 03:05:46.990377 17789 ts_tablet_manager.cc:1939] T e0f8960231a245ae9d0007ddb95587f9 P 441b104b3e3b405e835601a8682c25b0: Deleting consensus metadata
Sep 28 03:05:46 dist-test-slave-fjww krb5kdc[17243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:05:46.992131 17273 catalog_manager.cc:4841] TS 441b104b3e3b405e835601a8682c25b0 (127.6.192.2:37695): tablet e0f8960231a245ae9d0007ddb95587f9 (table test-table [id=43526846cf7e48e5bcd409428b889dc3]) successfully deleted
I20260928 03:05:47.000396 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:46.963612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.6.192.1:34819)
0928 03:05:46.963821 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:46.963827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:46.968156 (+ 4329us) server_negotiation.cc:407] Connection header received
0928 03:05:46.968496 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:46.968504 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:46.968641 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:46.968732 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:46.974717 (+ 5985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.976132 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.979830 (+ 3698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:46.980646 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:46.980827 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:46.993302 (+ 12475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:46.993317 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:46.993333 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:46.993380 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:46.996467 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:46.997551 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:46.997562 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:46.997565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:46.997744 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:46.998232 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:46.998236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:46.998237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:46.998627 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:46.998857 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:46.999384 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 03:05:46.999777 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 03:05:47.001206 17472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:47.098287 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 17349
I20260928 03:05:47.122681 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 17483
Sep 28 03:05:47 dist-test-slave-fjww krb5kdc[17243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.2@KRBTEST.COM
I20260928 03:05:47.143378 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:47.123610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58554 (local address 127.6.192.2:37695)
0928 03:05:47.123810 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:47.123818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:47.123907 (+ 89us) server_negotiation.cc:407] Connection header received
0928 03:05:47.124097 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:47.124104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:47.124203 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:47.124329 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:47.125656 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:47.127097 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:47.128590 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:47.129400 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:47.129565 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:47.135233 (+ 5668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:47.135259 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:47.135288 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:47.135359 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:47.140015 (+ 4656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:47.140905 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:47.140916 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:47.140920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:47.141137 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:47.141552 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:47.141558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:47.141561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:47.142093 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:47.142472 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:47.142680 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 03:05:47.142941 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 03:05:47.144331 17606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:47.259665 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 17483
I20260928 03:05:47.278317 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 17617
Sep 28 03:05:47 dist-test-slave-fjww krb5kdc[17243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.3@KRBTEST.COM
I20260928 03:05:47.294453 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:05:47.279130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.6.192.3:41387)
0928 03:05:47.279343 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 03:05:47.279347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:05:47.279367 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:05:47.279605 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:05:47.279610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:05:47.279733 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:05:47.279833 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:05:47.281239 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:47.281966 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:47.283200 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:05:47.283837 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:05:47.283943 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:05:47.288463 (+ 4520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:05:47.288480 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:05:47.288497 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:05:47.288589 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:05:47.291779 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:47.292472 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:47.292480 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:47.292482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:47.292731 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:05:47.293095 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:05:47.293098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:05:47.293100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:05:47.293458 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:05:47.293754 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 03:05:47.294016 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 03:05:47.294183 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 03:05:47.295202 17741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:47.387112 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 17617
I20260928 03:05:47.405928 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 17256
I20260928 03:05:47.407270 17318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:05:47.521756 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 17256
2026-09-28T03:05:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4592 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:05:47 dist-test-slave-fjww krb5kdc[17799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:05:47 dist-test-slave-fjww krb5kdc[17799](info): set up 2 sockets
Sep 28 03:05:47 dist-test-slave-fjww krb5kdc[17799](info): commencing operation
krb5kdc: starting...
W20260928 03:05:48.684525 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:05:48 dist-test-slave-fjww krb5kdc[17799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:05:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:05:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.192.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.192.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.192.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/rangeradmin_127.6.192.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.192.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/rangeradmin_127.6.192.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.192.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.192.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.192.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/rangerlookup_127.6.192.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.192.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/rangerlookup_127.6.192.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.192.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.192.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.192.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/HTTP_127.6.192.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.192.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/HTTP_127.6.192.20@KRBTEST.COM.keytab.
I20260928 03:05:49.046932 6912 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task97hosG/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/postgres -l logfile start
2026-09-28 03:05:51.881 UTC [17829] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 03:05:51.881 UTC [17829] LOG: listening on IPv4 address "127.6.192.20", port 45281
2026-09-28 03:05:51.885 UTC [17829] LOG: listening on Unix socket "/tmp/.s.PGSQL.45281"
2026-09-28 03:05:51.908 UTC [17832] LOG: database system was shut down at 2026-09-28 03:05:50 UTC
2026-09-28 03:05:51.912 UTC [17829] LOG: database system is ready to accept connections
I20260928 03:05:52.992697 6912 mini_postgres.cc:96] Postgres bound to 45281
2026-09-28 03:05:53.012 UTC [17840] FATAL: database "slave" does not exist
127.6.192.20:45281 - accepting connections
I20260928 03:05:53.013167 6912 mini_ranger.cc:161] Starting Ranger...
I20260928 03:05:53.044512 6912 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 03:05:53.608402 6912 mini_ranger.cc:87] Created ranger Postgres database
I20260928 03:05:53.608680 6912 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ranger-admin
2026-09-28 03:05:53,896 [I] DB FLAVOR :POSTGRES
2026-09-28 03:05:53,896 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 03:05:53,896 [I] Checking connection..
2026-09-28 03:05:53,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 03:05:54,188 [I] Checking connection passed.
2026-09-28 03:05:54,188 [I] --------- Verifying version history table ---------
2026-09-28 03:05:54,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 03:05:54,431 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 03:05:54,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 03:05:54,707 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 03:05:54,707 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 03:05:54,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 03:05:54.980 UTC [17904] WARNING: there is no transaction in progress
2026-09-28 03:05:54,987 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 03:05:54,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 03:05:55,265 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 03:05:55,265 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 03:05:55,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 03:05:55,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 03:05:55,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fjww.c.gcp-upstream.internal','N') ;"
2026-09-28 03:05:55,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 03:05:56,226 [I] Table x_portal_user does not exist in database ranger
2026-09-28 03:05:56,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 03:05:56,489 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 03:05:56,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 03:05:56,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 03:05:56,993 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 03:05:56,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 03:05:58.067 UTC [18012] WARNING: there is no transaction in progress
2026-09-28 03:05:58.085 UTC [18012] WARNING: there is no transaction in progress
2026-09-28 03:05:58.104 UTC [18012] WARNING: there is no transaction in progress
2026-09-28 03:05:58.124 UTC [18012] WARNING: there is no transaction in progress
2026-09-28 03:05:58.142 UTC [18012] WARNING: there is no transaction in progress
2026-09-28 03:05:58.162 UTC [18012] WARNING: there is no transaction in progress
2026-09-28 03:05:58.235 UTC [18012] WARNING: there is no transaction in progress
2026-09-28 03:05:58.244 UTC [18012] WARNING: there is no transaction in progress
2026-09-28 03:05:58.260 UTC [18012] WARNING: there is no transaction in progress
2026-09-28 03:05:58.272 UTC [18012] WARNING: there is no transaction in progress
2026-09-28 03:05:58.284 UTC [18012] WARNING: there is no transaction in progress
2026-09-28 03:05:58.301 UTC [18012] WARNING: there is no transaction in progress
2026-09-28 03:05:58.313 UTC [18012] WARNING: there is no transaction in progress
2026-09-28 03:05:58.322 UTC [18012] WARNING: there is no transaction in progress
2026-09-28 03:05:58.331 UTC [18012] WARNING: there is no transaction in progress
2026-09-28 03:05:58.339 UTC [18012] WARNING: there is no transaction in progress
2026-09-28 03:05:59,123 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 03:05:59,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 03:05:59,410 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 03:05:59,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 03:05:59,653 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 03:05:59,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 03:05:59,896 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 03:05:59,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 03:06:00,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 03:06:00,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fjww.c.gcp-upstream.internal';"
2026-09-28 03:06:00,633 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 03:06:00,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 03:06:00,890 [I] DB_PATCHES have already been applied
I20260928 03:06:00.896236 6912 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ranger-admin:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task97hosG/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 03:06:00.896368 6912 mini_ranger.cc:193] Using host: 127.6.192.20
I20260928 03:06:00.912972 6912 mini_ranger.cc:227] Ranger admin URL: http://127.6.192.20:59961
Sep 28, 2026 3:06:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:06:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:06:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 3:06:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 3:06:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 3:06:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 3:06:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 03:06:02.155838 6912 mini_ranger.cc:160] Time spent starting Ranger: real 9.143s user 0.005s sys 0.041s
Sep 28, 2026 3:06:23 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 3:06:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 3:06:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 3:06:23 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 3:06:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 3:06:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 3:06:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 3:06:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 3:06:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 3:06:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 3:06:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 3:06:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 3:06:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 3:06:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 3:06:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 3:06:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 3:06:23 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 3:06:24 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 03:06:25.424086 6912 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.192.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.192.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.192.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/rangerkms_127.6.192.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.192.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/rangerkms_127.6.192.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.192.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.192.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.192.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/HTTP_127.6.192.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.192.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/HTTP_127.6.192.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 03:06:25 dist-test-slave-fjww krb5kdc[17799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564785, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 03:06:25.626754 6912 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 03:06:25.652330 6912 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 03:06:26.674783 6912 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 03:06:26.674960 6912 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ranger-kms
I20260928 03:06:26.675107 6912 mini_ranger_kms.cc:227] Using postgres at 127.6.192.20:45281
2026-09-28 03:06:27,128 [I] DB FLAVOR :POSTGRES
2026-09-28 03:06:27,128 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 03:06:27,128 [I] Checking connection
2026-09-28 03:06:27,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 03:06:27,383 [I] connection success
2026-09-28 03:06:27,384 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 03:06:27,384 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 03:06:27,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 03:06:27,667 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 03:06:27,667 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 03:06:27,667 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 03:06:27,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:45281/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 03:06:27,987 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 03:06:28.534876 6912 mini_ranger_kms.cc:310] Created kms service
I20260928 03:06:28.766721 6912 mini_ranger_kms.cc:326] Created kudu user
I20260928 03:06:28.821022 6912 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 03:06:28 dist-test-slave-fjww krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564785, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
I20260928 03:06:29.269349 6912 mini_ranger_kms.cc:384] Added ranger policy
I20260928 03:06:29.269706 6912 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ranger-kms:/tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 03:06:29.269778 6912 mini_ranger_kms.cc:240] Using host: 127.6.192.20
I20260928 03:06:29.288571 6912 mini_ranger_kms.cc:276] Ranger KMS PID: 18193
I20260928 03:06:29.288808 6912 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.192.20:55523
Sep 28, 2026 3:06:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:06:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:06:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 3:06:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 3:06:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 3:06:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 03:06:31 dist-test-slave-fjww krb5kdc[17799](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790564791, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.192.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:06:31.608760 6912 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.982s user 0.159s sys 0.320s
I20260928 03:06:31.609023 6912 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 03:06:31.609278 6912 mini_ranger_kms.cc:401] 127.6.192.20:55523/kms/v1/keys
Sep 28 03:06:32 dist-test-slave-fjww krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564791, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.192.20@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
Sep 28 03:06:32 dist-test-slave-fjww krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564791, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.192.20@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
Sep 28 03:06:34 dist-test-slave-fjww krb5kdc[17799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:06:35.020349 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:34881
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:34881
--ranger_config_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.192.20:55523/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:06:35.376506 18243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:06:35.376827 18243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:06:35.376977 18243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:06:35.384696 18243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:06:35.384828 18243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:06:35.384907 18243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:06:35.384992 18243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:06:35.385059 18243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:06:35.395627 18243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.192.20:55523/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:34881
--ranger_config_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:34881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:06:35.397121 18243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:06:35.398512 18243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:06:35.412437 18248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:06:35.451918 18251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:06:35.452015 18249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:06:35.452003 18243 server_base.cc:1034] running on GCE node
I20260928 03:06:35.453087 18243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:06:35.454715 18243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:06:35.455940 18243 hybrid_clock.cc:648] HybridClock initialized: now 1790564795455865 us; error 71 us; skew 500 ppm
I20260928 03:06:35.456449 18243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:06:35 dist-test-slave-fjww krb5kdc[17799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564795, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:06:35.460216 18243 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:06:35.462039 18243 webserver.cc:460] Webserver started at http://127.6.192.62:41447/ using document root <none> and password file <none>
I20260928 03:06:35.462790 18243 fs_manager.cc:359] Metadata directory not provided
I20260928 03:06:35.462896 18243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:06:35.463233 18243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 03:06:35 dist-test-slave-fjww krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564795, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
I20260928 03:06:35.535039 18243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "d5391e850afa4ea5909a1120e2212f48"
format_stamp: "Formatted at 2026-09-28 03:06:35 on dist-test-slave-fjww"
server_key: "6f9ccb9e2434d19b8c65da1519495520"
server_key_iv: "92f957fc4a038fbe6691490952c1bd5c"
server_key_version: "kuduclusterkey@0"
I20260928 03:06:35.536046 18243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "d5391e850afa4ea5909a1120e2212f48"
format_stamp: "Formatted at 2026-09-28 03:06:35 on dist-test-slave-fjww"
server_key: "6f9ccb9e2434d19b8c65da1519495520"
server_key_iv: "92f957fc4a038fbe6691490952c1bd5c"
server_key_version: "kuduclusterkey@0"
I20260928 03:06:35.542009 18243 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260928 03:06:35.562726 18258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:06:35.564424 18243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:06:35.564631 18243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "d5391e850afa4ea5909a1120e2212f48"
format_stamp: "Formatted at 2026-09-28 03:06:35 on dist-test-slave-fjww"
server_key: "6f9ccb9e2434d19b8c65da1519495520"
server_key_iv: "92f957fc4a038fbe6691490952c1bd5c"
server_key_version: "kuduclusterkey@0"
I20260928 03:06:35.564857 18243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:06:35.586966 18243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:06:35.592514 18243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:06:35.592878 18243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:06:35.618883 18243 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:34881
I20260928 03:06:35.618896 18309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:34881 every 8 connection(s)
I20260928 03:06:35.620443 18243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:06:35.621495 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 18243
I20260928 03:06:35.621729 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:06:35.622730 6912 external_mini_cluster.cc:1428] Setting key 45b6e1b40e1efbb1a64ff03f33637f0a
I20260928 03:06:35.625533 18310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:06:35.638975 18310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48: Bootstrap starting.
I20260928 03:06:35.642721 18310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48: Neither blocks nor log segments found. Creating new log.
I20260928 03:06:35.644176 18310 log.cc:824] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48: Log is configured to *not* fsync() on all Append() calls
I20260928 03:06:35.647765 18310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48: No bootstrap required, opened a new log
Sep 28 03:06:35 dist-test-slave-fjww krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:06:35.659287 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:35.631151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48208 (local address 127.6.192.62:34881)
0928 03:06:35.632884 (+ 1733us) server_negotiation.cc:206] Beginning negotiation
0928 03:06:35.632908 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 03:06:35.636162 (+ 3254us) server_negotiation.cc:407] Connection header received
0928 03:06:35.638141 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:06:35.638165 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:06:35.638586 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:06:35.639006 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:06:35.640627 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:35.642035 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:35.643984 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:35.645216 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:35.645375 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:35.649881 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:06:35.649897 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:06:35.649919 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:06:35.649970 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:06:35.654095 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:06:35.655135 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:06:35.655144 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:06:35.655154 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:06:35.655338 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:06:35.655808 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:06:35.655813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:06:35.655816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:06:35.656619 (+ 803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:06:35.656967 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0928 03:06:35.657365 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 03:06:35.657840 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1406,"thread_start_us":1120,"threads_started":1}
I20260928 03:06:35.659353 18310 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5391e850afa4ea5909a1120e2212f48" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34881 } }
I20260928 03:06:35.659883 18310 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:06:35.660023 18310 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5391e850afa4ea5909a1120e2212f48, State: Initialized, Role: FOLLOWER
I20260928 03:06:35.660665 18310 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5391e850afa4ea5909a1120e2212f48" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34881 } }
I20260928 03:06:35.660931 18310 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:06:35.661077 18310 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:06:35.661221 18310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:06:35.663769 18310 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5391e850afa4ea5909a1120e2212f48" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34881 } }
I20260928 03:06:35.664310 18310 leader_election.cc:302] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5391e850afa4ea5909a1120e2212f48; no voters:
I20260928 03:06:35.664940 18310 leader_election.cc:288] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:06:35.665104 18315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:06:35.665954 18315 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48 [term 1 LEADER]: Becoming Leader. State: Replica: d5391e850afa4ea5909a1120e2212f48, State: Running, Role: LEADER
I20260928 03:06:35.666738 18315 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5391e850afa4ea5909a1120e2212f48" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34881 } }
I20260928 03:06:35.667619 18310 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:06:35.669909 18317 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5391e850afa4ea5909a1120e2212f48. Latest consensus state: current_term: 1 leader_uuid: "d5391e850afa4ea5909a1120e2212f48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5391e850afa4ea5909a1120e2212f48" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34881 } } }
I20260928 03:06:35.670413 18317 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48 [sys.catalog]: This master's current role is: LEADER
I20260928 03:06:35.671327 18316 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5391e850afa4ea5909a1120e2212f48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5391e850afa4ea5909a1120e2212f48" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34881 } } }
I20260928 03:06:35.671808 18316 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48 [sys.catalog]: This master's current role is: LEADER
I20260928 03:06:35.672091 18321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:06:35.677652 18321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:06:35.682054 18310 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 03:06:35.697839 18321 catalog_manager.cc:1347] Generated new cluster ID: dbca760fa0fe4518a64149f0435c04bf
I20260928 03:06:35.698040 18321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:06:35.714954 18321 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:06:35.717588 18321 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:06:35.735299 18321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48: Generated new TSK 0
I20260928 03:06:35.737146 18321 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
I20260928 03:06:36.492178 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.192.20:55523/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34881
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:06:36 dist-test-slave-fjww krb5kdc[17799](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790564796, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 03:06:36.934653 18356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:06:36.935014 18356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:06:36.935195 18356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:06:36.946066 18356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:06:36.946246 18356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:06:36.946491 18356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:06:36.959125 18356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.192.20:55523/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:06:36.960572 18356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:06:36.962006 18356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:06:36.990360 18366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:06:36.992378 18367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:06:37.004528 18369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:06:37.005314 18356 server_base.cc:1034] running on GCE node
I20260928 03:06:37.006150 18356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:06:37.007391 18356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:06:37.008735 18356 hybrid_clock.cc:648] HybridClock initialized: now 1790564797008650 us; error 69 us; skew 500 ppm
I20260928 03:06:37.009179 18356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:06:37 dist-test-slave-fjww krb5kdc[17799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564797, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:06:37.012759 18356 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:06:37.014814 18356 webserver.cc:460] Webserver started at http://127.6.192.1:37237/ using document root <none> and password file <none>
I20260928 03:06:37.015676 18356 fs_manager.cc:359] Metadata directory not provided
I20260928 03:06:37.015846 18356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:06:37.016196 18356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 03:06:37 dist-test-slave-fjww krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564797, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
I20260928 03:06:37.033993 18356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "dee79c459bd345a5ba5834c62798f684"
format_stamp: "Formatted at 2026-09-28 03:06:37 on dist-test-slave-fjww"
server_key: "d8d5461275d580dc93c06d20634ede58"
server_key_iv: "596a72426f0f28e163b4ea63836e8c1c"
server_key_version: "kuduclusterkey@0"
I20260928 03:06:37.035804 18356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "dee79c459bd345a5ba5834c62798f684"
format_stamp: "Formatted at 2026-09-28 03:06:37 on dist-test-slave-fjww"
server_key: "d8d5461275d580dc93c06d20634ede58"
server_key_iv: "596a72426f0f28e163b4ea63836e8c1c"
server_key_version: "kuduclusterkey@0"
I20260928 03:06:37.047428 18356 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.009s sys 0.000s
I20260928 03:06:37.080201 18376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:06:37.082576 18356 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.008s sys 0.008s
I20260928 03:06:37.082927 18356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "dee79c459bd345a5ba5834c62798f684"
format_stamp: "Formatted at 2026-09-28 03:06:37 on dist-test-slave-fjww"
server_key: "d8d5461275d580dc93c06d20634ede58"
server_key_iv: "596a72426f0f28e163b4ea63836e8c1c"
server_key_version: "kuduclusterkey@0"
I20260928 03:06:37.083328 18356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:06:37.159458 18356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:06:37.165552 18356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:06:37.165998 18356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:06:37.170639 18356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:06:37.173688 18356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:06:37.173874 18356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 03:06:37.174104 18356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:06:37.174247 18356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:06:37.254837 18356 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:41335
I20260928 03:06:37.254884 18488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:41335 every 8 connection(s)
I20260928 03:06:37.256762 18356 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:06:37.265220 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 18356
I20260928 03:06:37.265468 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:06:37.266520 6912 external_mini_cluster.cc:1428] Setting key f2ff6c385fffaaf6b9ea470a4964f472
Sep 28 03:06:37 dist-test-slave-fjww krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564797, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:06:37.291489 18492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:37.261287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:50527 (local address 127.6.192.62:34881)
0928 03:06:37.267167 (+ 5880us) server_negotiation.cc:206] Beginning negotiation
0928 03:06:37.267173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:06:37.267197 (+ 24us) server_negotiation.cc:407] Connection header received
0928 03:06:37.267266 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:06:37.267273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:06:37.267367 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:06:37.269271 (+ 1904us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:06:37.271674 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:37.272837 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:37.275412 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:37.276104 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:37.276205 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:37.281023 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:06:37.281031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:06:37.281036 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:06:37.281090 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:06:37.284872 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:06:37.285947 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:06:37.285953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:06:37.285957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:06:37.286120 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:06:37.286671 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:06:37.286676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:06:37.286679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:06:37.287027 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:06:37.287151 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 03:06:37.290703 (+ 3552us) server_negotiation.cc:299] Negotiation successful
0928 03:06:37.291045 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5733,"thread_start_us":5605,"threads_started":1}
I20260928 03:06:37.291951 18491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:37.260763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34881 (local address 127.6.192.1:50527)
0928 03:06:37.263271 (+ 2508us) negotiation.cc:107] Waiting for socket to connect
0928 03:06:37.263339 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 03:06:37.266001 (+ 2662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:06:37.269780 (+ 3779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:06:37.269795 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:06:37.270515 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:06:37.271407 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:06:37.271434 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:06:37.273057 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:06:37.273062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:06:37.275220 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:06:37.275246 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:06:37.276562 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:06:37.276569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:06:37.276876 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:37.277779 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:06:37.277810 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:06:37.280725 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:06:37.285066 (+ 4341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:06:37.285076 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:06:37.285098 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:06:37.285754 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:06:37.286256 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:06:37.286261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:06:37.286265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:06:37.286526 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:06:37.287279 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:06:37.287289 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:06:37.287811 (+ 522us) client_negotiation.cc:769] Sending connection context
0928 03:06:37.288175 (+ 364us) client_negotiation.cc:240] Negotiation successful
0928 03:06:37.288895 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2148,"thread_start_us":1888,"threads_started":1}
I20260928 03:06:37.297636 18489 heartbeater.cc:347] Connected to a master server at 127.6.192.62:34881
I20260928 03:06:37.298054 18489 heartbeater.cc:464] Registering TS with master...
I20260928 03:06:37.299072 18489 heartbeater.cc:511] Master 127.6.192.62:34881 requested a full tablet report, sending...
I20260928 03:06:37.302129 18275 ts_manager.cc:194] Registered new tserver with Master: dee79c459bd345a5ba5834c62798f684 (127.6.192.1:41335)
I20260928 03:06:37.304311 18275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:50527
WARNING: no policy specified for kudu/127.6.192.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.2@KRBTEST.COM" created.
Sep 28 03:06:37 dist-test-slave-fjww krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564796, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
I20260928 03:06:37.366233 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.192.20:55523/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34881
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:06:37 dist-test-slave-fjww krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564796, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
W20260928 03:06:37.812139 18495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:06:37.812671 18495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:06:37.812857 18495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:06:37.823839 18495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:06:37.824088 18495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:06:37.824299 18495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:06:37.838490 18495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.192.20:55523/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:06:37.840544 18495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:06:37.842536 18495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:06:37.876614 18502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:06:37.880089 18503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:06:37.881703 18495 server_base.cc:1034] running on GCE node
W20260928 03:06:37.881886 18505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:06:37.882647 18495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:06:37.895876 18495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:06:37.899711 18495 hybrid_clock.cc:648] HybridClock initialized: now 1790564797899618 us; error 101 us; skew 500 ppm
I20260928 03:06:37.900272 18495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:06:37 dist-test-slave-fjww krb5kdc[17799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564797, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:06:37.904973 18495 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:06:37.907258 18495 webserver.cc:460] Webserver started at http://127.6.192.2:34047/ using document root <none> and password file <none>
I20260928 03:06:37.908114 18495 fs_manager.cc:359] Metadata directory not provided
I20260928 03:06:37.908262 18495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:06:37.908624 18495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 03:06:37 dist-test-slave-fjww krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564797, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
I20260928 03:06:37.927611 18495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "b385d58a608343e3b6140a4e61dc46dd"
format_stamp: "Formatted at 2026-09-28 03:06:37 on dist-test-slave-fjww"
server_key: "a075832a4798be0b426c69176758c019"
server_key_iv: "53e7903af6698888fcd5a7a4f395726c"
server_key_version: "kuduclusterkey@0"
I20260928 03:06:37.928939 18495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "b385d58a608343e3b6140a4e61dc46dd"
format_stamp: "Formatted at 2026-09-28 03:06:37 on dist-test-slave-fjww"
server_key: "a075832a4798be0b426c69176758c019"
server_key_iv: "53e7903af6698888fcd5a7a4f395726c"
server_key_version: "kuduclusterkey@0"
I20260928 03:06:37.950145 18495 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.020s sys 0.000s
I20260928 03:06:37.972199 18519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:06:37.974334 18495 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.004s
I20260928 03:06:37.974552 18495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "b385d58a608343e3b6140a4e61dc46dd"
format_stamp: "Formatted at 2026-09-28 03:06:37 on dist-test-slave-fjww"
server_key: "a075832a4798be0b426c69176758c019"
server_key_iv: "53e7903af6698888fcd5a7a4f395726c"
server_key_version: "kuduclusterkey@0"
I20260928 03:06:37.974876 18495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:06:38.010494 18495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:06:38.016614 18495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:06:38.017048 18495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:06:38.018399 18495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:06:38.020538 18495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:06:38.020650 18495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:06:38.020839 18495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:06:38.020918 18495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:06:38.058701 18495 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:34031
I20260928 03:06:38.058804 18631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:34031 every 8 connection(s)
I20260928 03:06:38.060491 18495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:06:38.061864 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 18495
I20260928 03:06:38.062031 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:06:38.062575 6912 external_mini_cluster.cc:1428] Setting key 8a5fa9006db294216846433d4d72ea33
Sep 28 03:06:38 dist-test-slave-fjww krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564797, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:06:38.086169 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:38.063472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:34937 (local address 127.6.192.62:34881)
0928 03:06:38.063960 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0928 03:06:38.063966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:06:38.065563 (+ 1597us) server_negotiation.cc:407] Connection header received
0928 03:06:38.067785 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:06:38.067790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:06:38.067893 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:06:38.068030 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:06:38.069892 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.070633 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.072402 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.072838 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.072971 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:38.077139 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:06:38.077149 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:06:38.077158 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:06:38.077209 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:06:38.081765 (+ 4556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:06:38.082947 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:06:38.082954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:06:38.082958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:06:38.083156 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:06:38.083851 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:06:38.083857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:06:38.083861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:06:38.084303 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:06:38.084467 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 03:06:38.085484 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0928 03:06:38.085747 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":241,"threads_started":1}
I20260928 03:06:38.087592 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:38.064218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34881 (local address 127.6.192.2:34937)
0928 03:06:38.065415 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0928 03:06:38.065460 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 03:06:38.067304 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:06:38.068294 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:06:38.068304 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:06:38.068986 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:06:38.069712 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:06:38.069735 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.070806 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.070811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:06:38.072252 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:06:38.072264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.073075 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.073082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:06:38.073342 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:38.074078 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:06:38.074101 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:06:38.076871 (+ 2770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:06:38.081989 (+ 5118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:06:38.082002 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:06:38.082025 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:06:38.082756 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:06:38.083326 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:06:38.083335 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:06:38.083341 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:06:38.083684 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:06:38.084468 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:06:38.084479 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:06:38.085008 (+ 529us) client_negotiation.cc:769] Sending connection context
0928 03:06:38.085335 (+ 327us) client_negotiation.cc:240] Negotiation successful
0928 03:06:38.085954 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":633,"threads_started":1}
I20260928 03:06:38.089325 18632 heartbeater.cc:347] Connected to a master server at 127.6.192.62:34881
I20260928 03:06:38.089670 18632 heartbeater.cc:464] Registering TS with master...
I20260928 03:06:38.090570 18632 heartbeater.cc:511] Master 127.6.192.62:34881 requested a full tablet report, sending...
I20260928 03:06:38.092511 18275 ts_manager.cc:194] Registered new tserver with Master: b385d58a608343e3b6140a4e61dc46dd (127.6.192.2:34031)
WARNING: no policy specified for kudu/127.6.192.3@KRBTEST.COM; defaulting to no policy
I20260928 03:06:38.093796 18275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:34937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
I20260928 03:06:38.128433 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.192.20:55523/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34881
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:06:38.309024 18489 heartbeater.cc:503] Master 127.6.192.62:34881 was elected leader, sending a full tablet report...
W20260928 03:06:38.487016 18638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:06:38.487406 18638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:06:38.487696 18638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:06:38.495939 18638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:06:38.496117 18638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:06:38.496255 18638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:06:38.506686 18638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.192.20:55523/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:34881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:06:38.508204 18638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:06:38.509582 18638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:06:38.524309 18643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:06:38.523991 18646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:06:38.523991 18644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:06:38.525866 18638 server_base.cc:1034] running on GCE node
I20260928 03:06:38.526556 18638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:06:38.527804 18638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:06:38.529049 18638 hybrid_clock.cc:648] HybridClock initialized: now 1790564798529001 us; error 43 us; skew 500 ppm
I20260928 03:06:38.529434 18638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:06:38 dist-test-slave-fjww krb5kdc[17799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564798, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:06:38.533653 18638 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:06:38.535648 18638 webserver.cc:460] Webserver started at http://127.6.192.3:41063/ using document root <none> and password file <none>
I20260928 03:06:38.536463 18638 fs_manager.cc:359] Metadata directory not provided
I20260928 03:06:38.536651 18638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:06:38.537006 18638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 03:06:38 dist-test-slave-fjww krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564798, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
I20260928 03:06:38.556128 18638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "dd50b8a1184a4a0c9858aa5041a88b21"
format_stamp: "Formatted at 2026-09-28 03:06:38 on dist-test-slave-fjww"
server_key: "861c108d91be4ccc9679c69ae6653b2a"
server_key_iv: "e4aed22e40e3b6aee230ffcad1819a62"
server_key_version: "kuduclusterkey@0"
I20260928 03:06:38.557437 18638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "dd50b8a1184a4a0c9858aa5041a88b21"
format_stamp: "Formatted at 2026-09-28 03:06:38 on dist-test-slave-fjww"
server_key: "861c108d91be4ccc9679c69ae6653b2a"
server_key_iv: "e4aed22e40e3b6aee230ffcad1819a62"
server_key_version: "kuduclusterkey@0"
I20260928 03:06:38.564386 18638 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260928 03:06:38.582088 18653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:06:38.583784 18638 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 03:06:38.583992 18638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "dd50b8a1184a4a0c9858aa5041a88b21"
format_stamp: "Formatted at 2026-09-28 03:06:38 on dist-test-slave-fjww"
server_key: "861c108d91be4ccc9679c69ae6653b2a"
server_key_iv: "e4aed22e40e3b6aee230ffcad1819a62"
server_key_version: "kuduclusterkey@0"
I20260928 03:06:38.584237 18638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:06:38.597548 18638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:06:38.602099 18638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:06:38.602450 18638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:06:38.603703 18638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:06:38.605314 18638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:06:38.605402 18638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:06:38.605542 18638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:06:38.605607 18638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:06:38.646023 18638 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:40099
I20260928 03:06:38.646059 18765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:40099 every 8 connection(s)
I20260928 03:06:38.647604 18638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:06:38.657075 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 18638
I20260928 03:06:38.657281 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:06:38.657887 6912 external_mini_cluster.cc:1428] Setting key ac363aa7bb9466e6bc53ecb0cc4f1100
Sep 28 03:06:38 dist-test-slave-fjww krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564798, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:06:38.671319 18768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:38.650512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:52791 (local address 127.6.192.62:34881)
0928 03:06:38.651172 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0928 03:06:38.651179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:06:38.652447 (+ 1268us) server_negotiation.cc:407] Connection header received
0928 03:06:38.654200 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:06:38.654205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:06:38.654289 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:06:38.654403 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:06:38.656045 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.656853 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.659339 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.659859 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.660031 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:38.664251 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:06:38.664257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:06:38.664260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:06:38.664298 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:06:38.667562 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:06:38.668581 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:06:38.668592 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:06:38.668595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:06:38.668808 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:06:38.669408 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:06:38.669414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:06:38.669417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:06:38.669703 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:06:38.669880 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 03:06:38.670670 (+ 790us) server_negotiation.cc:299] Negotiation successful
0928 03:06:38.670911 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":286,"threads_started":1}
I20260928 03:06:38.672570 18769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:38.651385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34881 (local address 127.6.192.3:52791)
0928 03:06:38.652277 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0928 03:06:38.652316 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 03:06:38.653917 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:06:38.654619 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:06:38.654628 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:06:38.655089 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:06:38.655857 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:06:38.655877 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.657044 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.657051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:06:38.659115 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:06:38.659136 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.660204 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.660214 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:06:38.660496 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:38.661264 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:06:38.661297 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:06:38.663989 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:06:38.667765 (+ 3776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:06:38.667773 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:06:38.667788 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:06:38.668373 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:06:38.668946 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:06:38.668956 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:06:38.668959 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:06:38.669286 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:06:38.669838 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:06:38.669847 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:06:38.670296 (+ 449us) client_negotiation.cc:769] Sending connection context
0928 03:06:38.670573 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 03:06:38.671146 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":362,"threads_started":1}
I20260928 03:06:38.674244 18766 heartbeater.cc:347] Connected to a master server at 127.6.192.62:34881
I20260928 03:06:38.674572 18766 heartbeater.cc:464] Registering TS with master...
I20260928 03:06:38.675360 18766 heartbeater.cc:511] Master 127.6.192.62:34881 requested a full tablet report, sending...
I20260928 03:06:38.677052 18274 ts_manager.cc:194] Registered new tserver with Master: dd50b8a1184a4a0c9858aa5041a88b21 (127.6.192.3:40099)
I20260928 03:06:38.678085 18274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:52791
I20260928 03:06:38.687740 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:06:38.708463 18768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:38.693543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.6.192.62:34881)
0928 03:06:38.693744 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 03:06:38.693750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:06:38.694469 (+ 719us) server_negotiation.cc:407] Connection header received
0928 03:06:38.694626 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:06:38.694630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:06:38.694725 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:06:38.694841 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:06:38.696826 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.697576 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.699372 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.699820 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.699929 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:38.702659 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:06:38.702664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:06:38.702668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:06:38.702706 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:06:38.705596 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:06:38.706364 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:06:38.706368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:06:38.706371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:06:38.706518 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:06:38.706960 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:06:38.706965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:06:38.706967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:06:38.707309 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:06:38.707541 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 03:06:38.707958 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 03:06:38.708205 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 03:06:38.713873 18274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:06:38.722693 18274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:06:38.750945 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:38.738088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41335 (local address 127.0.0.1:47326)
0928 03:06:38.740584 (+ 2496us) negotiation.cc:107] Waiting for socket to connect
0928 03:06:38.740621 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 03:06:38.740819 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:06:38.741652 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:06:38.741660 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:06:38.741691 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:06:38.742173 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:06:38.742200 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.744261 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.744266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:06:38.747280 (+ 3014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:06:38.747303 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.749548 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.749556 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:06:38.749823 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:38.749886 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 03:06:38.750047 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 03:06:38.750327 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2332,"mutex_wait_us":52,"thread_start_us":2169,"threads_started":1}
I20260928 03:06:38.751261 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:38.737796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:34031 (local address 127.0.0.1:59908)
0928 03:06:38.738851 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0928 03:06:38.738893 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 03:06:38.739136 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:06:38.743005 (+ 3869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:06:38.743010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:06:38.743042 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:06:38.743502 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:06:38.743515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.745557 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.745563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:06:38.748403 (+ 2840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:06:38.748419 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.750310 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.750316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:06:38.750487 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:38.750527 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 03:06:38.750602 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 03:06:38.750754 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":57,"thread_start_us":497,"threads_started":1}
I20260928 03:06:38.751916 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:38.737908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.6.192.2:34031)
0928 03:06:38.742363 (+ 4455us) server_negotiation.cc:206] Beginning negotiation
0928 03:06:38.742375 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:06:38.742401 (+ 26us) server_negotiation.cc:407] Connection header received
0928 03:06:38.742520 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:06:38.742529 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:06:38.742783 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:06:38.743003 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:06:38.743709 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.745360 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.748620 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.749994 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.751015 (+ 1021us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:38.751268 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 03:06:38.751372 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 03:06:38.751593 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4256,"thread_start_us":4323,"threads_started":1}
I20260928 03:06:38.751929 18782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:38.738895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.6.192.1:41335)
0928 03:06:38.740003 (+ 1108us) server_negotiation.cc:206] Beginning negotiation
0928 03:06:38.740017 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:06:38.740982 (+ 965us) server_negotiation.cc:407] Connection header received
0928 03:06:38.741112 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:06:38.741125 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:06:38.741406 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:06:38.741644 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:06:38.742403 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.744068 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.747558 (+ 3490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.749135 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.749280 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:38.749560 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 03:06:38.750131 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 03:06:38.750333 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":654,"threads_started":1}
I20260928 03:06:38.753059 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:38.739624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:40099 (local address 127.0.0.1:49832)
0928 03:06:38.740960 (+ 1336us) negotiation.cc:107] Waiting for socket to connect
0928 03:06:38.740977 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:06:38.741094 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:06:38.742506 (+ 1412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:06:38.742510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:06:38.742541 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:06:38.742843 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:06:38.742853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.745160 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.745165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:06:38.747303 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:06:38.747343 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.749746 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.749753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:06:38.749922 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:38.749949 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 03:06:38.750046 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 03:06:38.750327 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1173,"mutex_wait_us":65,"thread_start_us":1002,"threads_started":1}
I20260928 03:06:38.753086 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:38.740413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.6.192.3:40099)
0928 03:06:38.741613 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0928 03:06:38.741624 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:06:38.741661 (+ 37us) server_negotiation.cc:407] Connection header received
0928 03:06:38.741780 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:06:38.741788 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:06:38.742050 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:06:38.742266 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:06:38.743110 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.744978 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.747593 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.749304 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.749438 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:38.749774 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 03:06:38.750821 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0928 03:06:38.751038 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":995,"thread_start_us":1572,"threads_started":1}
I20260928 03:06:38.756115 18424 tablet_service.cc:1467] Processing CreateTablet for tablet fb9e5c0cf3b2468e98f050c0b36929e5 (DEFAULT_TABLE table=test-table [id=457e675f342d446e82426c835148cae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:06:38.756155 18567 tablet_service.cc:1467] Processing CreateTablet for tablet fb9e5c0cf3b2468e98f050c0b36929e5 (DEFAULT_TABLE table=test-table [id=457e675f342d446e82426c835148cae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:06:38.757573 18424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9e5c0cf3b2468e98f050c0b36929e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:06:38.757613 18567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9e5c0cf3b2468e98f050c0b36929e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:06:38.757447 18701 tablet_service.cc:1467] Processing CreateTablet for tablet fb9e5c0cf3b2468e98f050c0b36929e5 (DEFAULT_TABLE table=test-table [id=457e675f342d446e82426c835148cae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:06:38.758870 18701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9e5c0cf3b2468e98f050c0b36929e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:06:38.771646 18785 tablet_bootstrap.cc:492] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684: Bootstrap starting.
I20260928 03:06:38.776368 18787 tablet_bootstrap.cc:492] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21: Bootstrap starting.
I20260928 03:06:38.776726 18785 tablet_bootstrap.cc:654] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684: Neither blocks nor log segments found. Creating new log.
I20260928 03:06:38.778527 18785 log.cc:824] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684: Log is configured to *not* fsync() on all Append() calls
I20260928 03:06:38.780242 18786 tablet_bootstrap.cc:492] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd: Bootstrap starting.
I20260928 03:06:38.780449 18787 tablet_bootstrap.cc:654] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21: Neither blocks nor log segments found. Creating new log.
I20260928 03:06:38.781791 18787 log.cc:824] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21: Log is configured to *not* fsync() on all Append() calls
I20260928 03:06:38.782445 18785 tablet_bootstrap.cc:492] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684: No bootstrap required, opened a new log
I20260928 03:06:38.782763 18785 ts_tablet_manager.cc:1397] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 03:06:38.784752 18786 tablet_bootstrap.cc:654] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd: Neither blocks nor log segments found. Creating new log.
I20260928 03:06:38.786506 18786 log.cc:824] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd: Log is configured to *not* fsync() on all Append() calls
I20260928 03:06:38.791597 18787 tablet_bootstrap.cc:492] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21: No bootstrap required, opened a new log
I20260928 03:06:38.791967 18787 ts_tablet_manager.cc:1397] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260928 03:06:38.792047 18785 raft_consensus.cc:348] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b385d58a608343e3b6140a4e61dc46dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 34031 } } peers { permanent_uuid: "dee79c459bd345a5ba5834c62798f684" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41335 } } peers { permanent_uuid: "dd50b8a1184a4a0c9858aa5041a88b21" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40099 } }
I20260928 03:06:38.792601 18785 raft_consensus.cc:374] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:06:38.792747 18785 raft_consensus.cc:729] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee79c459bd345a5ba5834c62798f684, State: Initialized, Role: FOLLOWER
I20260928 03:06:38.793452 18785 consensus_queue.cc:260] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b385d58a608343e3b6140a4e61dc46dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 34031 } } peers { permanent_uuid: "dee79c459bd345a5ba5834c62798f684" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41335 } } peers { permanent_uuid: "dd50b8a1184a4a0c9858aa5041a88b21" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40099 } }
I20260928 03:06:38.794708 18786 tablet_bootstrap.cc:492] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd: No bootstrap required, opened a new log
I20260928 03:06:38.795018 18786 ts_tablet_manager.cc:1397] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260928 03:06:38.802840 18787 raft_consensus.cc:348] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b385d58a608343e3b6140a4e61dc46dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 34031 } } peers { permanent_uuid: "dee79c459bd345a5ba5834c62798f684" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41335 } } peers { permanent_uuid: "dd50b8a1184a4a0c9858aa5041a88b21" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40099 } }
I20260928 03:06:38.803344 18787 raft_consensus.cc:374] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:06:38.803493 18787 raft_consensus.cc:729] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd50b8a1184a4a0c9858aa5041a88b21, State: Initialized, Role: FOLLOWER
I20260928 03:06:38.804354 18787 consensus_queue.cc:260] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b385d58a608343e3b6140a4e61dc46dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 34031 } } peers { permanent_uuid: "dee79c459bd345a5ba5834c62798f684" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41335 } } peers { permanent_uuid: "dd50b8a1184a4a0c9858aa5041a88b21" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40099 } }
I20260928 03:06:38.805081 18785 ts_tablet_manager.cc:1428] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684: Time spent starting tablet: real 0.022s user 0.006s sys 0.015s
I20260928 03:06:38.807837 18786 raft_consensus.cc:348] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b385d58a608343e3b6140a4e61dc46dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 34031 } } peers { permanent_uuid: "dee79c459bd345a5ba5834c62798f684" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41335 } } peers { permanent_uuid: "dd50b8a1184a4a0c9858aa5041a88b21" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40099 } }
I20260928 03:06:38.808358 18786 raft_consensus.cc:374] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:06:38.808486 18786 raft_consensus.cc:729] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b385d58a608343e3b6140a4e61dc46dd, State: Initialized, Role: FOLLOWER
I20260928 03:06:38.809194 18786 consensus_queue.cc:260] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b385d58a608343e3b6140a4e61dc46dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 34031 } } peers { permanent_uuid: "dee79c459bd345a5ba5834c62798f684" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41335 } } peers { permanent_uuid: "dd50b8a1184a4a0c9858aa5041a88b21" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40099 } }
I20260928 03:06:38.812429 18632 heartbeater.cc:503] Master 127.6.192.62:34881 was elected leader, sending a full tablet report...
I20260928 03:06:38.813450 18786 ts_tablet_manager.cc:1428] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260928 03:06:38.814225 18787 ts_tablet_manager.cc:1428] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21: Time spent starting tablet: real 0.022s user 0.015s sys 0.006s
I20260928 03:06:38.814628 18766 heartbeater.cc:503] Master 127.6.192.62:34881 was elected leader, sending a full tablet report...
I20260928 03:06:38.895804 18792 raft_consensus.cc:482] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:06:38.896389 18792 raft_consensus.cc:504] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b385d58a608343e3b6140a4e61dc46dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 34031 } } peers { permanent_uuid: "dee79c459bd345a5ba5834c62798f684" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41335 } } peers { permanent_uuid: "dd50b8a1184a4a0c9858aa5041a88b21" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40099 } }
I20260928 03:06:38.898253 18792 leader_election.cc:288] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b385d58a608343e3b6140a4e61dc46dd (127.6.192.2:34031), dee79c459bd345a5ba5834c62798f684 (127.6.192.1:41335)
W20260928 03:06:38.899803 18767 tablet.cc:2369] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:06:38.905138 18769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:38.898601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41335 (local address 127.6.192.3:46239)
0928 03:06:38.898808 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 03:06:38.898834 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 03:06:38.899035 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:06:38.900005 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:06:38.900010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:06:38.900036 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:06:38.900264 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:06:38.900278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.901703 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.901711 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:06:38.903094 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:06:38.903107 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.904356 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.904362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:06:38.904552 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:38.904576 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 03:06:38.904691 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 03:06:38.904845 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 03:06:38.905129 18782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:38.899036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:46239 (local address 127.6.192.1:41335)
0928 03:06:38.899247 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 03:06:38.899253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:06:38.899272 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:06:38.899350 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:06:38.899354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:06:38.899431 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:06:38.899561 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:06:38.900438 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.901532 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.903292 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.904187 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.904345 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:38.904454 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 03:06:38.904731 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 03:06:38.904869 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 03:06:38.905597 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:38.899184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:37373 (local address 127.6.192.2:34031)
0928 03:06:38.899400 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 03:06:38.899406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:06:38.899676 (+ 270us) server_negotiation.cc:407] Connection header received
0928 03:06:38.899948 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:06:38.899955 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:06:38.900037 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:06:38.900102 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:06:38.900963 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.901903 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.903799 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.904609 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.904778 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:38.904885 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 03:06:38.905199 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 03:06:38.905333 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 03:06:38.905711 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:38.898601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:34031 (local address 127.6.192.3:37373)
0928 03:06:38.899561 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0928 03:06:38.899577 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:06:38.899760 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:06:38.900291 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:06:38.900295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:06:38.900321 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:06:38.900773 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:06:38.900786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.902071 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.902076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:06:38.903614 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:06:38.903628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.904829 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.904836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:06:38.905017 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:38.905040 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 03:06:38.905163 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 03:06:38.905295 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":80,"thread_start_us":981,"threads_started":1}
I20260928 03:06:38.906473 18444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9e5c0cf3b2468e98f050c0b36929e5" candidate_uuid: "dd50b8a1184a4a0c9858aa5041a88b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee79c459bd345a5ba5834c62798f684" is_pre_election: true
I20260928 03:06:38.906680 18587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9e5c0cf3b2468e98f050c0b36929e5" candidate_uuid: "dd50b8a1184a4a0c9858aa5041a88b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b385d58a608343e3b6140a4e61dc46dd" is_pre_election: true
I20260928 03:06:38.907063 18444 raft_consensus.cc:2454] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd50b8a1184a4a0c9858aa5041a88b21 in term 0.
I20260928 03:06:38.907222 18587 raft_consensus.cc:2454] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd50b8a1184a4a0c9858aa5041a88b21 in term 0.
I20260928 03:06:38.907799 18657 leader_election.cc:302] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd50b8a1184a4a0c9858aa5041a88b21, dee79c459bd345a5ba5834c62798f684; no voters:
I20260928 03:06:38.908226 18792 raft_consensus.cc:2781] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:06:38.908389 18792 raft_consensus.cc:482] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:06:38.908526 18792 raft_consensus.cc:3037] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:06:38.910727 18792 raft_consensus.cc:504] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b385d58a608343e3b6140a4e61dc46dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 34031 } } peers { permanent_uuid: "dee79c459bd345a5ba5834c62798f684" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41335 } } peers { permanent_uuid: "dd50b8a1184a4a0c9858aa5041a88b21" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40099 } }
I20260928 03:06:38.911357 18792 leader_election.cc:288] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [CANDIDATE]: Term 1 election: Requested vote from peers b385d58a608343e3b6140a4e61dc46dd (127.6.192.2:34031), dee79c459bd345a5ba5834c62798f684 (127.6.192.1:41335)
I20260928 03:06:38.912039 18444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9e5c0cf3b2468e98f050c0b36929e5" candidate_uuid: "dd50b8a1184a4a0c9858aa5041a88b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee79c459bd345a5ba5834c62798f684"
I20260928 03:06:38.912070 18587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9e5c0cf3b2468e98f050c0b36929e5" candidate_uuid: "dd50b8a1184a4a0c9858aa5041a88b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b385d58a608343e3b6140a4e61dc46dd"
I20260928 03:06:38.912264 18444 raft_consensus.cc:3037] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:06:38.912241 18587 raft_consensus.cc:3037] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:06:38.914532 18444 raft_consensus.cc:2454] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd50b8a1184a4a0c9858aa5041a88b21 in term 1.
I20260928 03:06:38.914621 18587 raft_consensus.cc:2454] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd50b8a1184a4a0c9858aa5041a88b21 in term 1.
I20260928 03:06:38.915036 18657 leader_election.cc:302] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd50b8a1184a4a0c9858aa5041a88b21, dee79c459bd345a5ba5834c62798f684; no voters:
I20260928 03:06:38.915338 18792 raft_consensus.cc:2781] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:06:38.915884 18792 raft_consensus.cc:686] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [term 1 LEADER]: Becoming Leader. State: Replica: dd50b8a1184a4a0c9858aa5041a88b21, State: Running, Role: LEADER
I20260928 03:06:38.916359 18792 consensus_queue.cc:237] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b385d58a608343e3b6140a4e61dc46dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 34031 } } peers { permanent_uuid: "dee79c459bd345a5ba5834c62798f684" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41335 } } peers { permanent_uuid: "dd50b8a1184a4a0c9858aa5041a88b21" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40099 } }
I20260928 03:06:38.921386 18274 catalog_manager.cc:5495] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd50b8a1184a4a0c9858aa5041a88b21 (127.6.192.3). New cstate: current_term: 1 leader_uuid: "dd50b8a1184a4a0c9858aa5041a88b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b385d58a608343e3b6140a4e61dc46dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 34031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee79c459bd345a5ba5834c62798f684" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd50b8a1184a4a0c9858aa5041a88b21" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40099 } health_report { overall_health: HEALTHY } } }
I20260928 03:06:38.952241 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:38.946201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.6.192.3:40099)
0928 03:06:38.946385 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 03:06:38.946397 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:06:38.946415 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:06:38.946551 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:06:38.946555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:06:38.946654 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:06:38.946806 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:06:38.947257 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.948099 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.949811 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:38.950237 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:38.950345 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:38.950858 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:06:38.951671 (+ 813us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:06:38.951786 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 03:06:38.951953 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 03:06:38.952025 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 03:06:38.961112 18444 raft_consensus.cc:1261] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684 [term 1 FOLLOWER]: Refusing update from remote peer dd50b8a1184a4a0c9858aa5041a88b21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:06:38.961161 18587 raft_consensus.cc:1261] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd [term 1 FOLLOWER]: Refusing update from remote peer dd50b8a1184a4a0c9858aa5041a88b21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:06:38.962327 18795 consensus_queue.cc:1035] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b385d58a608343e3b6140a4e61dc46dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 34031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:06:38.962837 18792 consensus_queue.cc:1035] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee79c459bd345a5ba5834c62798f684" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:06:38.977633 18798 mvcc.cc:205] Tried to move back new op lower bound from 7334153416534265856 to 7334153416365203456. Current Snapshot: MvccSnapshot[applied={T|T < 7334153416534265856}]
I20260928 03:06:38.995924 18275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:06:38.996250 18275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:06:39.000346 18275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5391e850afa4ea5909a1120e2212f48: Sending DeleteTablet for 3 replicas of tablet fb9e5c0cf3b2468e98f050c0b36929e5
I20260928 03:06:39.001726 18424 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9e5c0cf3b2468e98f050c0b36929e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:06:38 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:47326
I20260928 03:06:39.001910 18701 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9e5c0cf3b2468e98f050c0b36929e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:06:38 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:49832
I20260928 03:06:39.002020 18567 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9e5c0cf3b2468e98f050c0b36929e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:06:38 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:59908
I20260928 03:06:39.004585 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 18356
I20260928 03:06:39.014482 18808 tablet_replica.cc:330] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684: stopping tablet replica
I20260928 03:06:39.015966 18808 raft_consensus.cc:2229] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:06:39.017580 18808 raft_consensus.cc:2258] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:06:39.021425 18808 ts_tablet_manager.cc:1905] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:06:39.026538 18809 tablet_replica.cc:330] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21: stopping tablet replica
I20260928 03:06:39.026803 18810 tablet_replica.cc:330] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd: stopping tablet replica
I20260928 03:06:39.027069 18809 raft_consensus.cc:2229] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:06:39.027290 18810 raft_consensus.cc:2229] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:06:39.027909 18810 raft_consensus.cc:2258] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:06:39.028787 18809 raft_consensus.cc:2258] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:06:39.030853 18810 ts_tablet_manager.cc:1905] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:06:39.030946 18809 ts_tablet_manager.cc:1905] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 03:06:39 dist-test-slave-fjww krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:06:39.037434 18810 ts_tablet_manager.cc:1918] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:06:39.037614 18810 log.cc:1196] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/fb9e5c0cf3b2468e98f050c0b36929e5
I20260928 03:06:39.038170 18810 ts_tablet_manager.cc:1939] T fb9e5c0cf3b2468e98f050c0b36929e5 P b385d58a608343e3b6140a4e61dc46dd: Deleting consensus metadata
I20260928 03:06:39.039399 18809 ts_tablet_manager.cc:1918] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:06:39.039633 18809 log.cc:1196] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/fb9e5c0cf3b2468e98f050c0b36929e5
I20260928 03:06:39.040232 18809 ts_tablet_manager.cc:1939] T fb9e5c0cf3b2468e98f050c0b36929e5 P dd50b8a1184a4a0c9858aa5041a88b21: Deleting consensus metadata
I20260928 03:06:39.042466 18808 ts_tablet_manager.cc:1918] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:06:39.042730 18808 log.cc:1196] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/fb9e5c0cf3b2468e98f050c0b36929e5
I20260928 03:06:39.042725 18260 catalog_manager.cc:4841] TS dd50b8a1184a4a0c9858aa5041a88b21 (127.6.192.3:40099): tablet fb9e5c0cf3b2468e98f050c0b36929e5 (table test-table [id=457e675f342d446e82426c835148cae7]) successfully deleted
I20260928 03:06:39.043318 18808 ts_tablet_manager.cc:1939] T fb9e5c0cf3b2468e98f050c0b36929e5 P dee79c459bd345a5ba5834c62798f684: Deleting consensus metadata
I20260928 03:06:39.044150 18260 catalog_manager.cc:4841] TS b385d58a608343e3b6140a4e61dc46dd (127.6.192.2:34031): tablet fb9e5c0cf3b2468e98f050c0b36929e5 (table test-table [id=457e675f342d446e82426c835148cae7]) successfully deleted
I20260928 03:06:39.045188 18262 catalog_manager.cc:4841] TS dee79c459bd345a5ba5834c62798f684 (127.6.192.1:41335): tablet fb9e5c0cf3b2468e98f050c0b36929e5 (table test-table [id=457e675f342d446e82426c835148cae7]) successfully deleted
I20260928 03:06:39.047629 18782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:39.006735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.6.192.1:41335)
0928 03:06:39.006987 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 03:06:39.006996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:06:39.020020 (+ 13024us) server_negotiation.cc:407] Connection header received
0928 03:06:39.020563 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:06:39.020571 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:06:39.020660 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:06:39.020738 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:06:39.022991 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:39.024099 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:39.027559 (+ 3460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:39.028343 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:39.028483 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:39.039133 (+ 10650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:06:39.039158 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:06:39.039178 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:06:39.039231 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:06:39.044110 (+ 4879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:06:39.045254 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:06:39.045263 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:06:39.045267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:06:39.045442 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:06:39.045970 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:06:39.045974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:06:39.045976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:06:39.046377 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:06:39.046604 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 03:06:39.047033 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 03:06:39.047234 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 03:06:39.048360 18484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:06:39.143463 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 18356
I20260928 03:06:39.169077 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 18495
Sep 28 03:06:39 dist-test-slave-fjww krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.2@KRBTEST.COM
I20260928 03:06:39.188429 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:39.169972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59914 (local address 127.6.192.2:34031)
0928 03:06:39.170182 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 03:06:39.170189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:06:39.170213 (+ 24us) server_negotiation.cc:407] Connection header received
0928 03:06:39.170439 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:06:39.170446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:06:39.170537 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:06:39.170675 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:06:39.172043 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:39.173319 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:39.174687 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:39.175407 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:39.175591 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:39.180345 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:06:39.180368 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:06:39.180391 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:06:39.180443 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:06:39.184908 (+ 4465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:06:39.185668 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:06:39.185679 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:06:39.185683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:06:39.185863 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:06:39.186365 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:06:39.186370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:06:39.186375 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:06:39.186895 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:06:39.187224 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 03:06:39.187737 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 03:06:39.187997 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 03:06:39.189246 18627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:06:39.280148 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 18495
I20260928 03:06:39.298511 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 18638
Sep 28 03:06:39 dist-test-slave-fjww krb5kdc[17799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.3@KRBTEST.COM
I20260928 03:06:39.317368 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:06:39.299529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49844 (local address 127.6.192.3:40099)
0928 03:06:39.299693 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 03:06:39.299698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:06:39.299716 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:06:39.299918 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:06:39.299922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:06:39.299982 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:06:39.300068 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:06:39.301814 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:39.302665 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:39.304709 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:06:39.305161 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:06:39.305291 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:06:39.310886 (+ 5595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:06:39.310902 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:06:39.310916 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:06:39.310960 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:06:39.314210 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:06:39.315062 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:06:39.315070 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:06:39.315072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:06:39.315208 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:06:39.315716 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:06:39.315720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:06:39.315722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:06:39.316114 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:06:39.316463 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0928 03:06:39.316764 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 03:06:39.316998 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 03:06:39.318061 18761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:06:39.440374 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 18638
I20260928 03:06:39.459653 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 18243
I20260928 03:06:39.460882 18305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:06:39.573669 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 18243
I20260928 03:06:39.595414 6912 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 03:06:39.931749 6912 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 03:06:39.932003 6912 mini_ranger.cc:66] Stopping Ranger...
I20260928 03:06:40.278640 6912 mini_ranger.cc:68] Stopped Ranger
2026-09-28 03:06:40.279 UTC [17829] LOG: received smart shutdown request
2026-09-28 03:06:40.283 UTC [17829] LOG: background worker "logical replication launcher" (PID 17838) exited with exit code 1
2026-09-28 03:06:40.283 UTC [17833] LOG: shutting down
2026-09-28 03:06:40.505 UTC [17829] LOG: database system is shut down
2026-09-28T03:06:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53115 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:06:40 dist-test-slave-fjww krb5kdc[18824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:06:40 dist-test-slave-fjww krb5kdc[18824](info): set up 2 sockets
Sep 28 03:06:40 dist-test-slave-fjww krb5kdc[18824](info): commencing operation
krb5kdc: starting...
W20260928 03:06:41.803964 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:06:41 dist-test-slave-fjww krb5kdc[18824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:06:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:06:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.192.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.192.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.192.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/rangeradmin_127.6.192.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.192.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/rangeradmin_127.6.192.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.192.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.192.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.192.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/rangerlookup_127.6.192.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.192.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/rangerlookup_127.6.192.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.192.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.192.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.192.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/HTTP_127.6.192.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.192.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/HTTP_127.6.192.20@KRBTEST.COM.keytab.
I20260928 03:06:42.163775 6912 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task97hosG/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/postgres -l logfile start
2026-09-28 03:06:45.034 UTC [18854] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 03:06:45.034 UTC [18854] LOG: listening on IPv4 address "127.6.192.20", port 39865
2026-09-28 03:06:45.039 UTC [18854] LOG: listening on Unix socket "/tmp/.s.PGSQL.39865"
2026-09-28 03:06:45.062 UTC [18857] LOG: database system was shut down at 2026-09-28 03:06:43 UTC
2026-09-28 03:06:45.067 UTC [18854] LOG: database system is ready to accept connections
I20260928 03:06:46.088218 6912 mini_postgres.cc:96] Postgres bound to 39865
2026-09-28 03:06:46.106 UTC [18865] FATAL: database "slave" does not exist
127.6.192.20:39865 - accepting connections
I20260928 03:06:46.107096 6912 mini_ranger.cc:161] Starting Ranger...
I20260928 03:06:46.136555 6912 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 03:06:46.708884 6912 mini_ranger.cc:87] Created ranger Postgres database
I20260928 03:06:46.709013 6912 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ranger-admin
2026-09-28 03:06:47,051 [I] DB FLAVOR :POSTGRES
2026-09-28 03:06:47,052 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 03:06:47,052 [I] Checking connection..
2026-09-28 03:06:47,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 03:06:47,301 [I] Checking connection passed.
2026-09-28 03:06:47,301 [I] --------- Verifying version history table ---------
2026-09-28 03:06:47,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 03:06:47,541 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 03:06:47,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 03:06:47,794 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 03:06:47,795 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 03:06:47,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 03:06:48.049 UTC [18929] WARNING: there is no transaction in progress
2026-09-28 03:06:48,056 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 03:06:48,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 03:06:48,328 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 03:06:48,329 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 03:06:48,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 03:06:48,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 03:06:48,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fjww.c.gcp-upstream.internal','N') ;"
2026-09-28 03:06:49,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 03:06:49,383 [I] Table x_portal_user does not exist in database ranger
2026-09-28 03:06:49,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 03:06:49,629 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 03:06:49,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 03:06:49,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 03:06:50,126 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 03:06:50,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 03:06:51.271 UTC [19038] WARNING: there is no transaction in progress
2026-09-28 03:06:51.290 UTC [19038] WARNING: there is no transaction in progress
2026-09-28 03:06:51.311 UTC [19038] WARNING: there is no transaction in progress
2026-09-28 03:06:51.329 UTC [19038] WARNING: there is no transaction in progress
2026-09-28 03:06:51.348 UTC [19038] WARNING: there is no transaction in progress
2026-09-28 03:06:51.371 UTC [19038] WARNING: there is no transaction in progress
2026-09-28 03:06:51.437 UTC [19038] WARNING: there is no transaction in progress
2026-09-28 03:06:51.445 UTC [19038] WARNING: there is no transaction in progress
2026-09-28 03:06:51.461 UTC [19038] WARNING: there is no transaction in progress
2026-09-28 03:06:51.474 UTC [19038] WARNING: there is no transaction in progress
2026-09-28 03:06:51.486 UTC [19038] WARNING: there is no transaction in progress
2026-09-28 03:06:51.503 UTC [19038] WARNING: there is no transaction in progress
2026-09-28 03:06:51.514 UTC [19038] WARNING: there is no transaction in progress
2026-09-28 03:06:51.522 UTC [19038] WARNING: there is no transaction in progress
2026-09-28 03:06:51.530 UTC [19038] WARNING: there is no transaction in progress
2026-09-28 03:06:51.537 UTC [19038] WARNING: there is no transaction in progress
2026-09-28 03:06:52,301 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 03:06:52,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 03:06:52,538 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 03:06:52,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 03:06:52,829 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 03:06:52,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 03:06:53,096 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 03:06:53,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 03:06:53,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 03:06:53,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fjww.c.gcp-upstream.internal';"
2026-09-28 03:06:53,852 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 03:06:53,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 03:06:54,115 [I] DB_PATCHES have already been applied
I20260928 03:06:54.121559 6912 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ranger-admin:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task97hosG/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 03:06:54.121650 6912 mini_ranger.cc:193] Using host: 127.6.192.20
I20260928 03:06:54.137264 6912 mini_ranger.cc:227] Ranger admin URL: http://127.6.192.20:33095
Sep 28, 2026 3:06:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:06:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:06:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 3:06:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 3:06:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 3:06:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 3:06:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 03:06:55.353734 6912 mini_ranger.cc:160] Time spent starting Ranger: real 9.247s user 0.000s sys 0.043s
Sep 28, 2026 3:07:17 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 3:07:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 3:07:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 3:07:17 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 3:07:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 3:07:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 3:07:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 3:07:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 3:07:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 3:07:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 3:07:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 3:07:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 3:07:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 3:07:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 3:07:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 3:07:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 3:07:17 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 3:07:18 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 03:07:19.229843 6912 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.192.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.192.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.192.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/rangerkms_127.6.192.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.192.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/rangerkms_127.6.192.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.192.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.192.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.192.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/HTTP_127.6.192.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.192.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/HTTP_127.6.192.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 03:07:19 dist-test-slave-fjww krb5kdc[18824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564839, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 03:07:19.416893 6912 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 03:07:19.439051 6912 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 03:07:20.495741 6912 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 03:07:20.495868 6912 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ranger-kms
I20260928 03:07:20.495909 6912 mini_ranger_kms.cc:227] Using postgres at 127.6.192.20:39865
2026-09-28 03:07:20,920 [I] DB FLAVOR :POSTGRES
2026-09-28 03:07:20,921 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 03:07:20,921 [I] Checking connection
2026-09-28 03:07:20,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 03:07:21,211 [I] connection success
2026-09-28 03:07:21,211 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 03:07:21,211 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 03:07:21,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 03:07:21,475 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 03:07:21,475 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 03:07:21,476 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 03:07:21,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.192.20:39865/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 03:07:21,764 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 03:07:22.312493 6912 mini_ranger_kms.cc:310] Created kms service
I20260928 03:07:22.561281 6912 mini_ranger_kms.cc:326] Created kudu user
I20260928 03:07:22.606312 6912 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 03:07:22 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564839, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
I20260928 03:07:23.053607 6912 mini_ranger_kms.cc:384] Added ranger policy
I20260928 03:07:23.053964 6912 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ranger-kms:/tmp/dist-test-task97hosG/build/asan/bin/postgresql.jar:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task97hosG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 03:07:23.054011 6912 mini_ranger_kms.cc:240] Using host: 127.6.192.20
I20260928 03:07:23.073673 6912 mini_ranger_kms.cc:276] Ranger KMS PID: 19219
I20260928 03:07:23.073853 6912 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.192.20:59997
Sep 28, 2026 3:07:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:07:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:07:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 3:07:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 3:07:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 3:07:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 03:07:25.539170 6912 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.122s user 0.101s sys 0.335s
I20260928 03:07:25.539424 6912 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 03:07:25.539618 6912 mini_ranger_kms.cc:401] 127.6.192.20:59997/kms/v1/keys
Sep 28 03:07:25 dist-test-slave-fjww krb5kdc[18824](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790564845, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.192.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 03:07:26 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564845, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.192.20@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
Sep 28 03:07:26 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564845, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.192.20@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
Sep 28 03:07:29 dist-test-slave-fjww krb5kdc[18824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:07:29.092854 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:41331
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:41331
--ranger_config_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.192.20:59997/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:07:29.445128 19269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:29.445501 19269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:29.445676 19269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:29.453357 19269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:07:29.453544 19269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:29.453647 19269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:29.453760 19269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:07:29.453857 19269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:07:29.464316 19269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.192.20:59997/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:41331
--ranger_config_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:41331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:29.465744 19269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:29.467217 19269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 03:07:29.480904 19269 server_base.cc:1034] running on GCE node
W20260928 03:07:29.480849 19277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:29.480873 19275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:29.481272 19274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:29.482172 19269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:29.483752 19269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:29.484984 19269 hybrid_clock.cc:648] HybridClock initialized: now 1790564849484939 us; error 34 us; skew 500 ppm
I20260928 03:07:29.485442 19269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:29 dist-test-slave-fjww krb5kdc[18824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564849, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:29.493927 19269 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:07:29.495810 19269 webserver.cc:460] Webserver started at http://127.6.192.62:33271/ using document root <none> and password file <none>
I20260928 03:07:29.496527 19269 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:29.496631 19269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:29.496910 19269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 03:07:29 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564849, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
I20260928 03:07:29.559887 19269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "809ac6bd2f8840b48323c598fdd9e37c"
format_stamp: "Formatted at 2026-09-28 03:07:29 on dist-test-slave-fjww"
server_key: "80426371762e079fa2ff5e4ad3857b1d"
server_key_iv: "0b3c07de4ace756a0ef8a904969ad67a"
server_key_version: "kuduclusterkey@0"
I20260928 03:07:29.560894 19269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "809ac6bd2f8840b48323c598fdd9e37c"
format_stamp: "Formatted at 2026-09-28 03:07:29 on dist-test-slave-fjww"
server_key: "80426371762e079fa2ff5e4ad3857b1d"
server_key_iv: "0b3c07de4ace756a0ef8a904969ad67a"
server_key_version: "kuduclusterkey@0"
I20260928 03:07:29.567533 19269 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260928 03:07:29.586937 19284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:29.589179 19269 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260928 03:07:29.589404 19269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "809ac6bd2f8840b48323c598fdd9e37c"
format_stamp: "Formatted at 2026-09-28 03:07:29 on dist-test-slave-fjww"
server_key: "80426371762e079fa2ff5e4ad3857b1d"
server_key_iv: "0b3c07de4ace756a0ef8a904969ad67a"
server_key_version: "kuduclusterkey@0"
I20260928 03:07:29.589715 19269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:29.613283 19269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:29.617779 19269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:29.618141 19269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:29.638919 19269 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:41331
I20260928 03:07:29.639025 19335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:41331 every 8 connection(s)
I20260928 03:07:29.640599 19269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:07:29.642611 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 19269
I20260928 03:07:29.642915 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:07:29.643680 6912 external_mini_cluster.cc:1428] Setting key aa68495b5c042db588d57460f9af5137
I20260928 03:07:29.645854 19336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:07:29.662822 19336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c: Bootstrap starting.
I20260928 03:07:29.667245 19336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c: Neither blocks nor log segments found. Creating new log.
I20260928 03:07:29.668958 19336 log.cc:824] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c: Log is configured to *not* fsync() on all Append() calls
Sep 28 03:07:29 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:07:29.672850 19336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c: No bootstrap required, opened a new log
I20260928 03:07:29.679682 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:29.646004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.6.192.62:41331)
0928 03:07:29.656259 (+ 10255us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:29.656273 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:29.656320 (+ 47us) server_negotiation.cc:407] Connection header received
0928 03:07:29.658081 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:29.658106 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:29.658569 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:29.658955 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:29.660986 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:29.662505 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:29.664620 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:29.665414 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:29.665582 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:29.671235 (+ 5653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:29.671244 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:29.671261 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:29.671304 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:29.674752 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:29.675911 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:29.675918 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:29.675925 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:29.676100 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:29.676599 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:29.676602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:29.676604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:29.677223 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:29.677473 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:29.678083 (+ 610us) server_negotiation.cc:299] Negotiation successful
0928 03:07:29.678417 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9964,"thread_start_us":9684,"threads_started":1}
I20260928 03:07:29.683982 19336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809ac6bd2f8840b48323c598fdd9e37c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41331 } }
I20260928 03:07:29.684474 19336 raft_consensus.cc:374] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:07:29.684607 19336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809ac6bd2f8840b48323c598fdd9e37c, State: Initialized, Role: FOLLOWER
I20260928 03:07:29.685290 19336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809ac6bd2f8840b48323c598fdd9e37c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41331 } }
I20260928 03:07:29.685602 19336 raft_consensus.cc:388] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:07:29.685791 19336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:07:29.686000 19336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:07:29.688753 19336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809ac6bd2f8840b48323c598fdd9e37c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41331 } }
I20260928 03:07:29.689503 19336 leader_election.cc:302] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 809ac6bd2f8840b48323c598fdd9e37c; no voters:
I20260928 03:07:29.690258 19336 leader_election.cc:288] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:07:29.690379 19341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:07:29.691907 19341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [term 1 LEADER]: Becoming Leader. State: Replica: 809ac6bd2f8840b48323c598fdd9e37c, State: Running, Role: LEADER
I20260928 03:07:29.692507 19341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809ac6bd2f8840b48323c598fdd9e37c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41331 } }
I20260928 03:07:29.693041 19336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:07:29.696275 19342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "809ac6bd2f8840b48323c598fdd9e37c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809ac6bd2f8840b48323c598fdd9e37c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41331 } } }
I20260928 03:07:29.696935 19342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [sys.catalog]: This master's current role is: LEADER
I20260928 03:07:29.696308 19343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 809ac6bd2f8840b48323c598fdd9e37c. Latest consensus state: current_term: 1 leader_uuid: "809ac6bd2f8840b48323c598fdd9e37c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809ac6bd2f8840b48323c598fdd9e37c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41331 } } }
I20260928 03:07:29.697845 19343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [sys.catalog]: This master's current role is: LEADER
I20260928 03:07:29.699777 19350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:07:29.705929 19350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:07:29.706782 19336 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 03:07:29.731494 19350 catalog_manager.cc:1347] Generated new cluster ID: f902d36153c944fdac1f323f7bff2161
I20260928 03:07:29.731820 19350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:07:29.743891 19350 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:07:29.746061 19350 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:07:29.759094 19350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c: Generated new TSK 0
I20260928 03:07:29.760403 19350 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
I20260928 03:07:30.551411 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.192.20:59997/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41331
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:07:30 dist-test-slave-fjww krb5kdc[18824](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790564850, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 03:07:31.011050 19381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:31.011545 19381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:31.011729 19381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:31.025460 19381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:31.025642 19381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:31.025952 19381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:07:31.044739 19381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.192.20:59997/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:31.046645 19381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:31.048655 19381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:31.068003 19391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:31.068049 19392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:31.081983 19394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:31.082258 19381 server_base.cc:1034] running on GCE node
I20260928 03:07:31.083364 19381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:31.084901 19381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:31.086215 19381 hybrid_clock.cc:648] HybridClock initialized: now 1790564851086138 us; error 73 us; skew 500 ppm
I20260928 03:07:31.086655 19381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:31 dist-test-slave-fjww krb5kdc[18824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564851, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:31.091368 19381 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:07:31.102208 19381 webserver.cc:460] Webserver started at http://127.6.192.1:40227/ using document root <none> and password file <none>
I20260928 03:07:31.103415 19381 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:31.103650 19381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:31.104147 19381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 03:07:31 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564851, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
I20260928 03:07:31.129556 19381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "7b468b468cee42338474bb5bf3055254"
format_stamp: "Formatted at 2026-09-28 03:07:31 on dist-test-slave-fjww"
server_key: "4a6510a07c9140b3d9b91042b2212dfc"
server_key_iv: "d1da4ab4f5a5dbd22c1a81cbf8df3922"
server_key_version: "kuduclusterkey@0"
I20260928 03:07:31.130483 19381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "7b468b468cee42338474bb5bf3055254"
format_stamp: "Formatted at 2026-09-28 03:07:31 on dist-test-slave-fjww"
server_key: "4a6510a07c9140b3d9b91042b2212dfc"
server_key_iv: "d1da4ab4f5a5dbd22c1a81cbf8df3922"
server_key_version: "kuduclusterkey@0"
I20260928 03:07:31.136451 19381 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 03:07:31.156836 19401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:31.159052 19381 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:07:31.159355 19381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "7b468b468cee42338474bb5bf3055254"
format_stamp: "Formatted at 2026-09-28 03:07:31 on dist-test-slave-fjww"
server_key: "4a6510a07c9140b3d9b91042b2212dfc"
server_key_iv: "d1da4ab4f5a5dbd22c1a81cbf8df3922"
server_key_version: "kuduclusterkey@0"
I20260928 03:07:31.159802 19381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:31.229552 19381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:31.235982 19381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:31.236449 19381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:31.237823 19381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:07:31.239961 19381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:07:31.240067 19381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:31.240278 19381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:07:31.240355 19381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:31.331255 19381 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:34389
I20260928 03:07:31.331310 19513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:34389 every 8 connection(s)
I20260928 03:07:31.333237 19381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:07:31.335927 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 19381
I20260928 03:07:31.336166 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:07:31.337076 6912 external_mini_cluster.cc:1428] Setting key 604f3a8a56bb6a99f3933a68980b07d6
Sep 28 03:07:31 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564850, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
Sep 28 03:07:31 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564851, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.192.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.2@KRBTEST.COM" created.
I20260928 03:07:31.420321 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:31.348180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:49653 (local address 127.6.192.62:41331)
0928 03:07:31.375313 (+ 27133us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:31.375321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:31.379515 (+ 4194us) server_negotiation.cc:407] Connection header received
0928 03:07:31.391646 (+ 12131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:31.391655 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:31.391804 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:31.391903 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:31.395961 (+ 4058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:31.397480 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:31.400616 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:31.401474 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:31.401605 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:31.410708 (+ 9103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:31.410716 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:31.410721 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:31.410814 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:31.415752 (+ 4938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:31.416926 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:31.416935 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:31.416942 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:31.417132 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:31.417789 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:31.417798 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:31.417804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:31.418275 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:31.418451 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:31.419427 (+ 976us) server_negotiation.cc:299] Negotiation successful
0928 03:07:31.419770 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26950,"thread_start_us":27144,"threads_started":1}
I20260928 03:07:31.421470 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:31.347427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41331 (local address 127.6.192.1:49653)
0928 03:07:31.376151 (+ 28724us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:31.376228 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:31.379197 (+ 2969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:31.393204 (+ 14007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:31.393224 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:31.394041 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:31.395013 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:31.395044 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:31.397750 (+ 2706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:31.397756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:31.400326 (+ 2570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:31.400348 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:31.402034 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:31.402044 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:31.402417 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:31.404281 (+ 1864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:31.404322 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:31.409181 (+ 4859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:31.415969 (+ 6788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:31.415982 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:31.416005 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:31.416760 (+ 755us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:31.417282 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:31.417288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:31.417293 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:31.417636 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:31.418436 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:31.418445 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:31.418952 (+ 507us) client_negotiation.cc:769] Sending connection context
0928 03:07:31.419302 (+ 350us) client_negotiation.cc:240] Negotiation successful
0928 03:07:31.419962 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28285,"thread_start_us":31941,"threads_started":1}
I20260928 03:07:31.423693 19514 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41331
I20260928 03:07:31.424064 19514 heartbeater.cc:464] Registering TS with master...
I20260928 03:07:31.424973 19514 heartbeater.cc:511] Master 127.6.192.62:41331 requested a full tablet report, sending...
I20260928 03:07:31.430020 19301 ts_manager.cc:194] Registered new tserver with Master: 7b468b468cee42338474bb5bf3055254 (127.6.192.1:34389)
I20260928 03:07:31.432368 19301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:49653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
W20260928 03:07:31.462980 6912 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.6.192.2: real 0.123s user 0.003s sys 0.016s
I20260928 03:07:31.464013 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.192.20:59997/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41331
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:07:31 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564850, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
W20260928 03:07:31.975939 19520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:31.976536 19520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:31.976809 19520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:31.990195 19520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:31.990476 19520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:31.990732 19520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:07:32.008567 19520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.192.20:59997/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:32.010785 19520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:32.012805 19520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:32.051739 19527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:32.055064 19528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:32.055586 19520 server_base.cc:1034] running on GCE node
W20260928 03:07:32.055728 19533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:32.057101 19520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:32.062952 19520 hybrid_clock.cc:648] HybridClock initialized: now 1790564852062863 us; error 85 us; skew 500 ppm
I20260928 03:07:32.063417 19520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:32 dist-test-slave-fjww krb5kdc[18824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564852, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:32.068205 19520 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:07:32.073228 19520 webserver.cc:460] Webserver started at http://127.6.192.2:45639/ using document root <none> and password file <none>
I20260928 03:07:32.074258 19520 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:32.074397 19520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:32.074808 19520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 03:07:32 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564852, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
I20260928 03:07:32.096724 19520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "0107b7cf7e214164847752340701bd3d"
format_stamp: "Formatted at 2026-09-28 03:07:32 on dist-test-slave-fjww"
server_key: "a805eb15914ff91f960cda2328710fac"
server_key_iv: "29015683cdb04a18c45bce0df124c6ef"
server_key_version: "kuduclusterkey@0"
I20260928 03:07:32.098186 19520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "0107b7cf7e214164847752340701bd3d"
format_stamp: "Formatted at 2026-09-28 03:07:32 on dist-test-slave-fjww"
server_key: "a805eb15914ff91f960cda2328710fac"
server_key_iv: "29015683cdb04a18c45bce0df124c6ef"
server_key_version: "kuduclusterkey@0"
I20260928 03:07:32.121948 19520 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.006s sys 0.011s
I20260928 03:07:32.145521 19544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:32.147719 19520 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260928 03:07:32.147996 19520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "0107b7cf7e214164847752340701bd3d"
format_stamp: "Formatted at 2026-09-28 03:07:32 on dist-test-slave-fjww"
server_key: "a805eb15914ff91f960cda2328710fac"
server_key_iv: "29015683cdb04a18c45bce0df124c6ef"
server_key_version: "kuduclusterkey@0"
I20260928 03:07:32.148355 19520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:32.231985 19520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:32.237859 19520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:32.238260 19520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:32.245800 19520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:07:32.254364 19520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:07:32.254585 19520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 03:07:32.254752 19520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:07:32.254873 19520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:32.436667 19514 heartbeater.cc:503] Master 127.6.192.62:41331 was elected leader, sending a full tablet report...
I20260928 03:07:32.562871 19520 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:33339
I20260928 03:07:32.562896 19657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:33339 every 8 connection(s)
I20260928 03:07:32.564240 19520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:07:32.565789 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 19520
I20260928 03:07:32.565975 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:07:32.566583 6912 external_mini_cluster.cc:1428] Setting key 822fc13fbb65d335bc26f009025b2586
Sep 28 03:07:32 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564852, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.192.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.3@KRBTEST.COM" created.
I20260928 03:07:32.606262 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:32.567320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:42381 (local address 127.6.192.62:41331)
0928 03:07:32.586554 (+ 19234us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:32.586559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:32.586575 (+ 16us) server_negotiation.cc:407] Connection header received
0928 03:07:32.586629 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:32.586633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:32.586734 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:32.586866 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:32.588946 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:32.590104 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:32.592856 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:32.593457 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:32.593583 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:32.598513 (+ 4930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:32.598520 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:32.598524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:32.598568 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:32.602468 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:32.603588 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:32.603593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:32.603596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:32.603757 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:32.604342 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:32.604348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:32.604351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:32.604739 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:32.604899 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:32.605644 (+ 745us) server_negotiation.cc:299] Negotiation successful
0928 03:07:32.605897 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19108,"thread_start_us":18967,"threads_started":1}
I20260928 03:07:32.607492 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:32.567994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41331 (local address 127.6.192.2:42381)
0928 03:07:32.582013 (+ 14019us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:32.582109 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:32.585560 (+ 3451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:32.587312 (+ 1752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:32.587322 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:32.587874 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:32.588700 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:32.588731 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:32.590293 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:32.590301 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:32.592665 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:32.592686 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:32.593718 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:32.593728 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:32.594096 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:32.595053 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:32.595089 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:32.598189 (+ 3100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:32.602679 (+ 4490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:32.602690 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:32.602712 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:32.603376 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:32.603944 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:32.603950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:32.603953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:32.604198 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:32.604881 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:32.604888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:32.605240 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 03:07:32.605475 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 03:07:32.605946 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13553,"thread_start_us":13254,"threads_started":1}
I20260928 03:07:32.609179 19658 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41331
I20260928 03:07:32.609515 19658 heartbeater.cc:464] Registering TS with master...
I20260928 03:07:32.610229 19658 heartbeater.cc:511] Master 127.6.192.62:41331 requested a full tablet report, sending...
I20260928 03:07:32.611871 19301 ts_manager.cc:194] Registered new tserver with Master: 0107b7cf7e214164847752340701bd3d (127.6.192.2:33339)
I20260928 03:07:32.612666 19301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:42381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
I20260928 03:07:32.638670 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.192.20:59997/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41331
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:07:32.989749 19664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:32.990047 19664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:32.990211 19664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:32.997941 19664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:32.998063 19664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:32.998235 19664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:07:33.008775 19664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.192.20:59997/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:33.010237 19664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:33.011664 19664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:33.026314 19669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:33.026264 19670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:33.026216 19672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:33.029249 19664 server_base.cc:1034] running on GCE node
I20260928 03:07:33.030005 19664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:33.031121 19664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:33.032366 19664 hybrid_clock.cc:648] HybridClock initialized: now 1790564853032296 us; error 70 us; skew 500 ppm
I20260928 03:07:33.032719 19664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:33 dist-test-slave-fjww krb5kdc[18824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564853, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:33.036170 19664 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:07:33.037746 19664 webserver.cc:460] Webserver started at http://127.6.192.3:39603/ using document root <none> and password file <none>
I20260928 03:07:33.038443 19664 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:33.038544 19664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:33.038808 19664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 03:07:33 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564853, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
I20260928 03:07:33.055532 19664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "134a6e21fa7543ae9c3b7099210a86b7"
format_stamp: "Formatted at 2026-09-28 03:07:33 on dist-test-slave-fjww"
server_key: "18f7429e40ccc58b38eefff6c76cdbc4"
server_key_iv: "f0a8a2f2d63bc8a3a9e0eeff8cf0a18f"
server_key_version: "kuduclusterkey@0"
I20260928 03:07:33.056438 19664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "134a6e21fa7543ae9c3b7099210a86b7"
format_stamp: "Formatted at 2026-09-28 03:07:33 on dist-test-slave-fjww"
server_key: "18f7429e40ccc58b38eefff6c76cdbc4"
server_key_iv: "f0a8a2f2d63bc8a3a9e0eeff8cf0a18f"
server_key_version: "kuduclusterkey@0"
I20260928 03:07:33.062377 19664 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 03:07:33.082300 19679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:33.084122 19664 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 03:07:33.084347 19664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "134a6e21fa7543ae9c3b7099210a86b7"
format_stamp: "Formatted at 2026-09-28 03:07:33 on dist-test-slave-fjww"
server_key: "18f7429e40ccc58b38eefff6c76cdbc4"
server_key_iv: "f0a8a2f2d63bc8a3a9e0eeff8cf0a18f"
server_key_version: "kuduclusterkey@0"
I20260928 03:07:33.084617 19664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:33.110018 19664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:33.114037 19664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:33.114380 19664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:33.115463 19664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:07:33.116972 19664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:07:33.117082 19664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:33.117244 19664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:07:33.117318 19664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:33.152709 19664 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:40777
I20260928 03:07:33.152750 19791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:40777 every 8 connection(s)
I20260928 03:07:33.154165 19664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:07:33.157116 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 19664
I20260928 03:07:33.157332 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:07:33.157891 6912 external_mini_cluster.cc:1428] Setting key 32dd68b46ae6efa112c4d5dced46f1ee
Sep 28 03:07:33 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564853, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:07:33.175725 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:33.157276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:58921 (local address 127.6.192.62:41331)
0928 03:07:33.157749 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:33.157755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:33.158953 (+ 1198us) server_negotiation.cc:407] Connection header received
0928 03:07:33.160980 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:33.160986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:33.161073 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:33.161164 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:33.162543 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:33.163228 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:33.164833 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:33.165353 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:33.165487 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:33.168931 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:33.168937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:33.168940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:33.168982 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:33.172038 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:33.173084 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:33.173092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:33.173097 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:33.173262 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:33.173878 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:33.173892 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:33.173896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:33.174275 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:33.174416 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:33.175111 (+ 695us) server_negotiation.cc:299] Negotiation successful
0928 03:07:33.175361 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":227,"threads_started":1}
I20260928 03:07:33.176631 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:33.157913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41331 (local address 127.6.192.3:58921)
0928 03:07:33.158788 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:33.158850 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:33.160720 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:33.161393 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:33.161401 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:33.161804 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:33.162381 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:33.162398 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:33.163363 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:33.163367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:33.164687 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:33.164699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:33.165558 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:33.165563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:33.165770 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:33.166489 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:33.166517 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:33.168652 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:33.172197 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:33.172210 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:33.172238 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:33.172900 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:33.173404 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:33.173407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:33.173410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:33.173650 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:33.174406 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:33.174413 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:33.174747 (+ 334us) client_negotiation.cc:769] Sending connection context
0928 03:07:33.174990 (+ 243us) client_negotiation.cc:240] Negotiation successful
0928 03:07:33.175380 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":368,"threads_started":1}
I20260928 03:07:33.178102 19792 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41331
I20260928 03:07:33.178435 19792 heartbeater.cc:464] Registering TS with master...
I20260928 03:07:33.179190 19792 heartbeater.cc:511] Master 127.6.192.62:41331 requested a full tablet report, sending...
I20260928 03:07:33.180701 19301 ts_manager.cc:194] Registered new tserver with Master: 134a6e21fa7543ae9c3b7099210a86b7 (127.6.192.3:40777)
I20260928 03:07:33.181515 19301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:58921
I20260928 03:07:33.186371 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:07:33.188040 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 19381
Sep 28 03:07:33 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:07:33.206460 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:33.189258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39712 (local address 127.6.192.1:34389)
0928 03:07:33.189961 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:33.189967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:33.189994 (+ 27us) server_negotiation.cc:407] Connection header received
0928 03:07:33.190051 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:33.190057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:33.190260 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:33.190383 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:33.192062 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:33.193123 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:33.194907 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:33.195537 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:33.195645 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:33.200496 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:33.200511 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:33.200528 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:33.200566 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:33.203623 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:33.204347 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:33.204354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:33.204356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:33.204501 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:33.204856 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:33.204859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:33.204861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:33.205186 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:33.205415 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:33.205829 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 03:07:33.206120 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":375,"threads_started":1}
I20260928 03:07:33.207782 19509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:33.293260 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 19381
I20260928 03:07:33.309896 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 19520
Sep 28 03:07:33 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.2@KRBTEST.COM
I20260928 03:07:33.332080 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:33.310948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52066 (local address 127.6.192.2:33339)
0928 03:07:33.311434 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:33.311494 (+ 60us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:33.311527 (+ 33us) server_negotiation.cc:407] Connection header received
0928 03:07:33.311609 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:33.311616 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:33.311830 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:33.311996 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:33.313423 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:33.314412 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:33.316056 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:33.316758 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:33.316865 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:33.326135 (+ 9270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:33.326154 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:33.326169 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:33.326215 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:33.329355 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:33.330136 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:33.330143 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:33.330145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:33.330286 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:33.330615 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:33.330618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:33.330620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:33.330973 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:33.331195 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:33.331523 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 03:07:33.331730 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":243,"threads_started":1}
I20260928 03:07:33.333422 19653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:33.417006 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 19520
I20260928 03:07:33.433094 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 19664
Sep 28 03:07:33 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.3@KRBTEST.COM
I20260928 03:07:33.450294 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:33.434055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.6.192.3:40777)
0928 03:07:33.434701 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:33.434708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:33.434739 (+ 31us) server_negotiation.cc:407] Connection header received
0928 03:07:33.434804 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:33.434810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:33.435042 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:33.435175 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:33.436447 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:33.437472 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:33.438711 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:33.439294 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:33.439402 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:33.444466 (+ 5064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:33.444484 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:33.444501 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:33.444543 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:33.447627 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:33.448418 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:33.448426 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:33.448428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:33.448562 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:33.448948 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:33.448951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:33.448953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:33.449307 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:33.449545 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:33.449772 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 03:07:33.449977 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":328,"threads_started":1}
I20260928 03:07:33.451730 19787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:33.535575 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 19664
I20260928 03:07:33.551322 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 19269
I20260928 03:07:33.552754 19331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:33.660805 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 19269
I20260928 03:07:33.683884 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:41331
--webserver_interface=127.6.192.62
--webserver_port=33271
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:41331
--ranger_config_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.192.20:59997/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.192.20:59997/kms with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:07:34.046059 19804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:34.046515 19804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:34.046707 19804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:34.054389 19804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:07:34.054524 19804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:34.054608 19804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:34.054705 19804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:07:34.054798 19804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:07:34.065549 19804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.192.20:59997/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:41331
--ranger_config_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:41331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=33271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:34.067029 19804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:34.068491 19804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:34.084072 19809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:34.085096 19810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:34.085355 19812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:34.086508 19804 server_base.cc:1034] running on GCE node
I20260928 03:07:34.087395 19804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:34.089035 19804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:34.090282 19804 hybrid_clock.cc:648] HybridClock initialized: now 1790564854090244 us; error 49 us; skew 500 ppm
I20260928 03:07:34.090667 19804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:34 dist-test-slave-fjww krb5kdc[18824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564854, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:34.094267 19804 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:07:34.095911 19804 webserver.cc:460] Webserver started at http://127.6.192.62:33271/ using document root <none> and password file <none>
I20260928 03:07:34.096634 19804 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:34.096788 19804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 03:07:34 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564854, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
I20260928 03:07:34.171274 19804 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 03:07:34.175999 19820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:34.178123 19804 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:07:34.178346 19804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "809ac6bd2f8840b48323c598fdd9e37c"
format_stamp: "Formatted at 2026-09-28 03:07:29 on dist-test-slave-fjww"
server_key: "80426371762e079fa2ff5e4ad3857b1d"
server_key_iv: "0b3c07de4ace756a0ef8a904969ad67a"
server_key_version: "kuduclusterkey@0"
I20260928 03:07:34.179419 19804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:34.198343 19804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:34.202754 19804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:34.203074 19804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:34.222833 19804 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:41331
I20260928 03:07:34.222891 19871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:41331 every 8 connection(s)
I20260928 03:07:34.224437 19804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:07:34.230409 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 19804
I20260928 03:07:34.231544 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.192.20:59997/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:34389
--local_ip_for_outbound_sockets=127.6.192.1
--tserver_master_addrs=127.6.192.62:41331
--webserver_port=40227
--webserver_interface=127.6.192.1
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.192.20:59997/kms with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:07:34.232173 19872 sys_catalog.cc:263] Verifying existing consensus state
I20260928 03:07:34.237808 19872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c: Bootstrap starting.
I20260928 03:07:34.257865 19872 log.cc:824] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c: Log is configured to *not* fsync() on all Append() calls
I20260928 03:07:34.267134 19872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 03:07:34.267776 19872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c: Bootstrap complete.
I20260928 03:07:34.277480 19872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809ac6bd2f8840b48323c598fdd9e37c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41331 } }
I20260928 03:07:34.278386 19872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809ac6bd2f8840b48323c598fdd9e37c, State: Initialized, Role: FOLLOWER
I20260928 03:07:34.279110 19872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809ac6bd2f8840b48323c598fdd9e37c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41331 } }
I20260928 03:07:34.279350 19872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:07:34.279505 19872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:07:34.279691 19872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [term 1 FOLLOWER]: Advancing to term 2
I20260928 03:07:34.283819 19872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809ac6bd2f8840b48323c598fdd9e37c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41331 } }
I20260928 03:07:34.284435 19872 leader_election.cc:302] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 809ac6bd2f8840b48323c598fdd9e37c; no voters:
I20260928 03:07:34.285138 19872 leader_election.cc:288] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 03:07:34.285413 19876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [term 2 FOLLOWER]: Leader election won for term 2
I20260928 03:07:34.287599 19872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:07:34.292369 19876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [term 2 LEADER]: Becoming Leader. State: Replica: 809ac6bd2f8840b48323c598fdd9e37c, State: Running, Role: LEADER
I20260928 03:07:34.293156 19876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809ac6bd2f8840b48323c598fdd9e37c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41331 } }
I20260928 03:07:34.298187 19878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 809ac6bd2f8840b48323c598fdd9e37c. Latest consensus state: current_term: 2 leader_uuid: "809ac6bd2f8840b48323c598fdd9e37c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809ac6bd2f8840b48323c598fdd9e37c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41331 } } }
I20260928 03:07:34.298600 19878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [sys.catalog]: This master's current role is: LEADER
I20260928 03:07:34.299180 19877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "809ac6bd2f8840b48323c598fdd9e37c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809ac6bd2f8840b48323c598fdd9e37c" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41331 } } }
I20260928 03:07:34.299503 19877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c [sys.catalog]: This master's current role is: LEADER
I20260928 03:07:34.309391 19886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:07:34.310127 19872 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 03:07:34.316505 19886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:07:34.323468 19886 catalog_manager.cc:1259] Loaded cluster ID: f902d36153c944fdac1f323f7bff2161
I20260928 03:07:34.324141 19886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:07:34.332016 19886 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:07:34.334316 19886 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c: Loaded TSK: 0
I20260928 03:07:34.336586 19886 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 03:07:34.706691 19874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:34.707248 19874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:34.707520 19874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:34.721103 19874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:34.721294 19874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:34.721592 19874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:07:34.740067 19874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.192.20:59997/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:34389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=40227
--tserver_master_addrs=127.6.192.62:41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:34.742002 19874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:34.744042 19874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:34.808421 19909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:34.823940 19910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:34.826787 19913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:34.827853 19874 server_base.cc:1034] running on GCE node
I20260928 03:07:34.829018 19874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:34.843812 19874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:34.845232 19874 hybrid_clock.cc:648] HybridClock initialized: now 1790564854845097 us; error 131 us; skew 500 ppm
I20260928 03:07:34.845778 19874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:34 dist-test-slave-fjww krb5kdc[18824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564854, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:34.854718 19874 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:07:34.867992 19874 webserver.cc:460] Webserver started at http://127.6.192.1:40227/ using document root <none> and password file <none>
I20260928 03:07:34.869220 19874 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:34.869493 19874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 03:07:34 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564854, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
I20260928 03:07:34.936601 19874 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.001s sys 0.005s
I20260928 03:07:34.941120 19921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:34.943163 19874 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260928 03:07:34.943431 19874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "7b468b468cee42338474bb5bf3055254"
format_stamp: "Formatted at 2026-09-28 03:07:31 on dist-test-slave-fjww"
server_key: "4a6510a07c9140b3d9b91042b2212dfc"
server_key_iv: "d1da4ab4f5a5dbd22c1a81cbf8df3922"
server_key_version: "kuduclusterkey@0"
I20260928 03:07:34.944950 19874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:34.980528 19874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:34.986509 19874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:34.986835 19874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:34.988968 19874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:07:34.990787 19874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:07:34.990885 19874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:34.991056 19874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:07:34.991161 19874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:35.094009 19874 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:34389
I20260928 03:07:35.094120 20033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:34389 every 8 connection(s)
I20260928 03:07:35.095885 19874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:07:35.100420 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 19874
I20260928 03:07:35.101302 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.192.20:59997/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:33339
--local_ip_for_outbound_sockets=127.6.192.2
--tserver_master_addrs=127.6.192.62:41331
--webserver_port=45639
--webserver_interface=127.6.192.2
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.192.20:59997/kms with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:07:35 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564854, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:07:35.152698 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:35.100154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41331 (local address 127.6.192.1:40651)
0928 03:07:35.114604 (+ 14450us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:35.114672 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:35.121701 (+ 7029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:35.127790 (+ 6089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:35.127808 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:35.128621 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:35.129699 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:35.129738 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:35.132900 (+ 3162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:35.132907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:35.135385 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:35.135407 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:35.136794 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:35.136804 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:35.137120 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:35.137963 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:35.137989 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:35.140902 (+ 2913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:35.146181 (+ 5279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:35.146192 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:35.146214 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:35.146964 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:35.147856 (+ 892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:35.147862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:35.147866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:35.148156 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:35.149354 (+ 1198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:35.149365 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:35.149924 (+ 559us) client_negotiation.cc:769] Sending connection context
0928 03:07:35.150388 (+ 464us) client_negotiation.cc:240] Negotiation successful
0928 03:07:35.151105 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14074,"thread_start_us":13851,"threads_started":1}
I20260928 03:07:35.152698 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:35.099970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:40651 (local address 127.6.192.62:41331)
0928 03:07:35.112172 (+ 12202us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:35.112197 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:35.114852 (+ 2655us) server_negotiation.cc:407] Connection header received
0928 03:07:35.122613 (+ 7761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:35.122633 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:35.123220 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:35.123760 (+ 540us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:35.130123 (+ 6363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:35.132596 (+ 2473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:35.135691 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:35.136471 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:35.136739 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:35.141311 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:35.141323 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:35.141341 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:35.141395 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:35.145929 (+ 4534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:35.147173 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:35.147184 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:35.147194 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:35.147515 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:35.148343 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:35.148349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:35.148352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:35.149045 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:35.149189 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:35.150736 (+ 1547us) server_negotiation.cc:299] Negotiation successful
0928 03:07:35.151191 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11847,"thread_start_us":12850,"threads_started":1}
I20260928 03:07:35.156940 20034 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41331
I20260928 03:07:35.157248 20034 heartbeater.cc:464] Registering TS with master...
I20260928 03:07:35.158175 20034 heartbeater.cc:511] Master 127.6.192.62:41331 requested a full tablet report, sending...
W20260928 03:07:35.161984 20034 heartbeater.cc:644] Failed to heartbeat to 127.6.192.62:41331 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 03:07:35.162245 20034 heartbeater.cc:464] Registering TS with master...
I20260928 03:07:35.162577 20034 heartbeater.cc:511] Master 127.6.192.62:41331 requested a full tablet report, sending...
I20260928 03:07:35.164230 20034 heartbeater.cc:464] Registering TS with master...
I20260928 03:07:35.164661 20034 heartbeater.cc:511] Master 127.6.192.62:41331 requested a full tablet report, sending...
W20260928 03:07:35.166002 20034 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 03:07:35.594699 20038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:35.595104 20038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:35.595304 20038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:35.607184 20038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:35.607345 20038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:35.607622 20038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:07:35.623716 20038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.192.20:59997/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:33339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=45639
--tserver_master_addrs=127.6.192.62:41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:35.625525 20038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:35.627247 20038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 28 03:07:35 dist-test-slave-fjww krb5kdc[18824](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790564855, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 03:07:35.647089 20054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:35.648062 20055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:35.654424 20057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:35.655530 20038 server_base.cc:1034] running on GCE node
I20260928 03:07:35.656414 20038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:35.660446 20038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:35.663910 20038 hybrid_clock.cc:648] HybridClock initialized: now 1790564855663827 us; error 68 us; skew 500 ppm
I20260928 03:07:35.664347 20038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:35 dist-test-slave-fjww krb5kdc[18824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564855, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:35.668826 20038 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:07:35.677112 20038 webserver.cc:460] Webserver started at http://127.6.192.2:45639/ using document root <none> and password file <none>
I20260928 03:07:35.678133 20038 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:35.678342 20038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 03:07:35 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564855, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
I20260928 03:07:35.704176 20038 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:07:35.707846 20066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:35.709661 20038 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 03:07:35.709860 20038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "0107b7cf7e214164847752340701bd3d"
format_stamp: "Formatted at 2026-09-28 03:07:32 on dist-test-slave-fjww"
server_key: "a805eb15914ff91f960cda2328710fac"
server_key_iv: "29015683cdb04a18c45bce0df124c6ef"
server_key_version: "kuduclusterkey@0"
I20260928 03:07:35.710938 20038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:35.742833 20038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:35.747989 20038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:35.748323 20038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:35.749629 20038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:07:35.751705 20038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:07:35.751880 20038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:35.752046 20038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:07:35.752167 20038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:35.801671 20038 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:33339
I20260928 03:07:35.801714 20178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:33339 every 8 connection(s)
I20260928 03:07:35.803416 20038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:07:35.806725 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 20038
I20260928 03:07:35.807622 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.192.20:59997/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:40777
--local_ip_for_outbound_sockets=127.6.192.3
--tserver_master_addrs=127.6.192.62:41331
--webserver_port=39603
--webserver_interface=127.6.192.3
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.192.20:59997/kms with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:07:35 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564855, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:07:35.860378 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:35.810328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:43045 (local address 127.6.192.62:41331)
0928 03:07:35.833865 (+ 23537us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:35.833872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:35.837943 (+ 4071us) server_negotiation.cc:407] Connection header received
0928 03:07:35.839647 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:35.839654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:35.839770 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:35.839859 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:35.842425 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:35.843616 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:35.846320 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:35.846998 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:35.847106 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:35.852230 (+ 5124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:35.852239 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:35.852243 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:35.852302 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:35.856122 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:35.857300 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:35.857307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:35.857312 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:35.857484 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:35.858067 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:35.858073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:35.858076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:35.858449 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:35.858577 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:35.859719 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0928 03:07:35.859996 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23365,"thread_start_us":23116,"threads_started":1}
I20260928 03:07:35.863229 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:35.811163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41331 (local address 127.6.192.2:43045)
0928 03:07:35.834495 (+ 23332us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:35.834555 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:35.839226 (+ 4671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:35.840521 (+ 1295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:35.840535 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:35.841257 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:35.842188 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:35.842217 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:35.843864 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:35.843869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:35.846127 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:35.846144 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:35.847513 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:35.847520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:35.847833 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:35.848673 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:35.848705 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:35.851644 (+ 2939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:35.856358 (+ 4714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:35.856369 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:35.856391 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:35.857139 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:35.857653 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:35.857659 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:35.857663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:35.857928 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:35.858697 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:35.858707 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:35.859198 (+ 491us) client_negotiation.cc:769] Sending connection context
0928 03:07:35.860984 (+ 1786us) client_negotiation.cc:240] Negotiation successful
0928 03:07:35.861620 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22955,"thread_start_us":26625,"threads_started":1}
I20260928 03:07:35.865914 20179 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41331
I20260928 03:07:35.866246 20179 heartbeater.cc:464] Registering TS with master...
I20260928 03:07:35.867110 20179 heartbeater.cc:511] Master 127.6.192.62:41331 requested a full tablet report, sending...
I20260928 03:07:35.869407 19837 ts_manager.cc:194] Registered new tserver with Master: 0107b7cf7e214164847752340701bd3d (127.6.192.2:33339)
I20260928 03:07:35.871711 19837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:43045
Sep 28 03:07:36 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564855, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
I20260928 03:07:36.166144 20034 heartbeater.cc:464] Registering TS with master...
I20260928 03:07:36.166592 20034 heartbeater.cc:511] Master 127.6.192.62:41331 requested a full tablet report, sending...
I20260928 03:07:36.167914 19837 ts_manager.cc:194] Registered new tserver with Master: 7b468b468cee42338474bb5bf3055254 (127.6.192.1:34389)
I20260928 03:07:36.169421 19837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:40651
W20260928 03:07:36.283877 20183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:36.284399 20183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:36.284615 20183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:36.297004 20183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:36.297320 20183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:36.297539 20183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:07:36.313721 20183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.192.20:59997/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:40777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=39603
--tserver_master_addrs=127.6.192.62:41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:36.315773 20183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:36.317634 20183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:36.360553 20189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:36.363394 20188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:36.367316 20183 server_base.cc:1034] running on GCE node
W20260928 03:07:36.369503 20192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:36.370505 20183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:36.371752 20183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:36.372990 20183 hybrid_clock.cc:648] HybridClock initialized: now 1790564856372918 us; error 64 us; skew 500 ppm
I20260928 03:07:36.373379 20183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:36 dist-test-slave-fjww krb5kdc[18824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564856, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:36.377039 20183 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:07:36.380918 20183 webserver.cc:460] Webserver started at http://127.6.192.3:39603/ using document root <none> and password file <none>
I20260928 03:07:36.381719 20183 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:36.381909 20183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 03:07:36 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564856, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
I20260928 03:07:36.419035 20183 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.007s sys 0.007s
I20260928 03:07:36.422502 20199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:36.424211 20183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 03:07:36.424443 20183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "134a6e21fa7543ae9c3b7099210a86b7"
format_stamp: "Formatted at 2026-09-28 03:07:33 on dist-test-slave-fjww"
server_key: "18f7429e40ccc58b38eefff6c76cdbc4"
server_key_iv: "f0a8a2f2d63bc8a3a9e0eeff8cf0a18f"
server_key_version: "kuduclusterkey@0"
I20260928 03:07:36.425557 20183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:36.457057 20183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:36.463200 20183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:36.463608 20183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:36.464813 20183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:07:36.482163 20183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:07:36.482321 20183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:36.482465 20183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:07:36.482549 20183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 03:07:36 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564855, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for HTTP/127.6.192.20@KRBTEST.COM
I20260928 03:07:36.702464 20183 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:40777
I20260928 03:07:36.702518 20312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:40777 every 8 connection(s)
I20260928 03:07:36.704145 20183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:07:36.711588 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 20183
Sep 28 03:07:36 dist-test-slave-fjww krb5kdc[18824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564856, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:07:36.751641 20318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:36.722231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47276 (local address 127.6.192.62:41331)
0928 03:07:36.731783 (+ 9552us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:36.731790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:36.731808 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:07:36.731874 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:36.731880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:36.731981 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:36.732102 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:36.734337 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.735578 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.739468 (+ 3890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.740145 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.740288 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:36.743842 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:36.743872 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:36.743880 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:36.743939 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:36.748050 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:36.749065 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:36.749070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:36.749073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:36.749259 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:36.749735 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:36.749739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:36.749742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:36.750212 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:36.750559 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:36.750879 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 03:07:36.751204 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9413,"thread_start_us":10028,"threads_started":1}
I20260928 03:07:36.753502 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:36.710500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:51529 (local address 127.6.192.62:41331)
0928 03:07:36.722261 (+ 11761us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:36.722266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:36.727527 (+ 5261us) server_negotiation.cc:407] Connection header received
0928 03:07:36.729735 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:36.729742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:36.729907 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:36.730023 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:36.733059 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.734236 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.736925 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.737557 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.737661 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:36.744395 (+ 6734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:36.744405 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:36.744410 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:36.744477 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:36.748970 (+ 4493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:36.750081 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:36.750086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:36.750090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:36.750279 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:36.750857 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:36.750861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:36.750865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:36.751246 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:36.751376 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:36.752640 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0928 03:07:36.752887 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11641,"thread_start_us":11508,"threads_started":1}
I20260928 03:07:36.754542 20315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:36.709600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41331 (local address 127.6.192.3:51529)
0928 03:07:36.723972 (+ 14372us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:36.724055 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:36.729049 (+ 4994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:36.730960 (+ 1911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:36.730978 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:36.731816 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:36.732807 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:36.732854 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.734488 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.734495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:36.736724 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:36.736741 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.738298 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.738306 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:36.738856 (+ 550us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:36.740808 (+ 1952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:36.740857 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:36.744034 (+ 3177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:36.749174 (+ 5140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:36.749182 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:36.749199 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:36.749948 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:36.750428 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:36.750433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:36.750436 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:36.750717 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:36.751627 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:36.751636 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:36.752180 (+ 544us) client_negotiation.cc:769] Sending connection context
0928 03:07:36.752534 (+ 354us) client_negotiation.cc:240] Negotiation successful
0928 03:07:36.753052 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13897,"thread_start_us":17786,"threads_started":1}
I20260928 03:07:36.756328 20313 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41331
I20260928 03:07:36.756664 20313 heartbeater.cc:464] Registering TS with master...
I20260928 03:07:36.757550 20313 heartbeater.cc:511] Master 127.6.192.62:41331 requested a full tablet report, sending...
I20260928 03:07:36.759579 19837 ts_manager.cc:194] Registered new tserver with Master: 134a6e21fa7543ae9c3b7099210a86b7 (127.6.192.3:40777)
I20260928 03:07:36.760715 19837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:51529
I20260928 03:07:36.767908 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:07:36.786999 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:36.771884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47290 (local address 127.6.192.62:41331)
0928 03:07:36.772156 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:36.772166 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:36.772726 (+ 560us) server_negotiation.cc:407] Connection header received
0928 03:07:36.772893 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:36.772897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:36.772967 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:36.773062 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:36.774644 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.775510 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.777034 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.777549 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.777673 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:36.780515 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:36.780520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:36.780524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:36.780562 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:36.783546 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:36.784362 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:36.784365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:36.784368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:36.784573 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:36.785099 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:36.785108 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:36.785110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:36.785461 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:36.785659 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:36.786165 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 03:07:36.786460 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 03:07:36.792595 19837 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:07:36.801741 19837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:07:36.839793 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:36.818356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:40777 (local address 127.0.0.1:41596)
0928 03:07:36.828659 (+ 10303us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:36.828708 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:36.828890 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:36.831354 (+ 2464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:36.831363 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:36.831423 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:07:36.831909 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:36.831922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.834544 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.834549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:36.836815 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:36.836829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.838645 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.838651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:36.838880 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:36.838952 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 03:07:36.839096 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 03:07:36.839320 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10019,"mutex_wait_us":23,"thread_start_us":9847,"threads_started":1}
I20260928 03:07:36.839815 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:36.818933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.6.192.3:40777)
0928 03:07:36.830611 (+ 11678us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:36.830621 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:36.830659 (+ 38us) server_negotiation.cc:407] Connection header received
0928 03:07:36.830756 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:36.830764 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:36.831031 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:36.831624 (+ 593us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:07:36.832153 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.834358 (+ 2205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.837023 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.838419 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.838541 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:36.838749 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:36.839196 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 03:07:36.839388 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11444,"thread_start_us":11224,"threads_started":1}
I20260928 03:07:36.840365 20338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:36.820016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52070 (local address 127.6.192.2:33339)
0928 03:07:36.820815 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:36.820829 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:36.829712 (+ 8883us) server_negotiation.cc:407] Connection header received
0928 03:07:36.830172 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:36.830184 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:36.830545 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:36.830789 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:07:36.832535 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.833793 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.836793 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.838382 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.838498 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:36.838749 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:36.839386 (+ 637us) server_negotiation.cc:299] Negotiation successful
0928 03:07:36.839591 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":350,"threads_started":1}
I20260928 03:07:36.841022 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:36.818934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:33339 (local address 127.0.0.1:52070)
0928 03:07:36.829568 (+ 10634us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:36.829591 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:36.829864 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:36.831139 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:36.831148 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:36.831182 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:07:36.832287 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:36.832301 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.833961 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.833965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:36.836592 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:36.836608 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.838953 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.838958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:36.839172 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:36.839205 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 03:07:36.840638 (+ 1433us) client_negotiation.cc:240] Negotiation successful
0928 03:07:36.840747 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10490,"mutex_wait_us":79,"thread_start_us":10331,"threads_started":1}
I20260928 03:07:36.842895 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:36.830991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:34389 (local address 127.0.0.1:39720)
0928 03:07:36.835044 (+ 4053us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:36.835060 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:36.835236 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:36.835925 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:36.835930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:36.835956 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:07:36.836387 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:36.836396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.838239 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.838244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:36.840414 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:36.840431 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.842048 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.842054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:36.842223 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:36.842264 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 03:07:36.842379 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 03:07:36.842507 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3928,"mutex_wait_us":4,"thread_start_us":3745,"threads_started":1}
I20260928 03:07:36.843616 20339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:36.829452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39720 (local address 127.6.192.1:34389)
0928 03:07:36.830959 (+ 1507us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:36.830968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:36.835184 (+ 4216us) server_negotiation.cc:407] Connection header received
0928 03:07:36.835428 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:36.835435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:36.835735 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:36.835945 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:07:36.836554 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.838099 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.840623 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:36.841824 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:36.841976 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:36.842227 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:36.842424 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 03:07:36.842584 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1317,"thread_start_us":1118,"threads_started":1}
I20260928 03:07:36.844588 20248 tablet_service.cc:1467] Processing CreateTablet for tablet 9df5c26c63114fa7bb8fea4cefbc2184 (DEFAULT_TABLE table=test-table [id=764a3104ff11440989f0eefc0676c673]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:07:36.844704 20114 tablet_service.cc:1467] Processing CreateTablet for tablet 9df5c26c63114fa7bb8fea4cefbc2184 (DEFAULT_TABLE table=test-table [id=764a3104ff11440989f0eefc0676c673]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:07:36.846160 20248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df5c26c63114fa7bb8fea4cefbc2184. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:07:36.846179 20114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df5c26c63114fa7bb8fea4cefbc2184. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:07:36.846390 19969 tablet_service.cc:1467] Processing CreateTablet for tablet 9df5c26c63114fa7bb8fea4cefbc2184 (DEFAULT_TABLE table=test-table [id=764a3104ff11440989f0eefc0676c673]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:07:36.847430 19969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df5c26c63114fa7bb8fea4cefbc2184. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:07:36.862118 20341 tablet_bootstrap.cc:492] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254: Bootstrap starting.
I20260928 03:07:36.865219 20343 tablet_bootstrap.cc:492] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7: Bootstrap starting.
I20260928 03:07:36.868646 20342 tablet_bootstrap.cc:492] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d: Bootstrap starting.
I20260928 03:07:36.870997 20343 tablet_bootstrap.cc:654] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7: Neither blocks nor log segments found. Creating new log.
I20260928 03:07:36.871398 20341 tablet_bootstrap.cc:654] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254: Neither blocks nor log segments found. Creating new log.
I20260928 03:07:36.872455 20342 tablet_bootstrap.cc:654] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d: Neither blocks nor log segments found. Creating new log.
I20260928 03:07:36.872460 20343 log.cc:824] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7: Log is configured to *not* fsync() on all Append() calls
I20260928 03:07:36.872756 20341 log.cc:824] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254: Log is configured to *not* fsync() on all Append() calls
I20260928 03:07:36.873766 20342 log.cc:824] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d: Log is configured to *not* fsync() on all Append() calls
I20260928 03:07:36.875802 20343 tablet_bootstrap.cc:492] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7: No bootstrap required, opened a new log
I20260928 03:07:36.876106 20343 ts_tablet_manager.cc:1397] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260928 03:07:36.876852 20342 tablet_bootstrap.cc:492] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d: No bootstrap required, opened a new log
I20260928 03:07:36.877156 20342 ts_tablet_manager.cc:1397] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 03:07:36.877274 20341 tablet_bootstrap.cc:492] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254: No bootstrap required, opened a new log
I20260928 03:07:36.877560 20341 ts_tablet_manager.cc:1397] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.006s
I20260928 03:07:36.881119 20179 heartbeater.cc:503] Master 127.6.192.62:41331 was elected leader, sending a full tablet report...
I20260928 03:07:36.888624 20343 raft_consensus.cc:348] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134a6e21fa7543ae9c3b7099210a86b7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40777 } } peers { permanent_uuid: "0107b7cf7e214164847752340701bd3d" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33339 } } peers { permanent_uuid: "7b468b468cee42338474bb5bf3055254" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34389 } }
I20260928 03:07:36.889183 20343 raft_consensus.cc:374] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:07:36.889292 20343 raft_consensus.cc:729] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 134a6e21fa7543ae9c3b7099210a86b7, State: Initialized, Role: FOLLOWER
I20260928 03:07:36.889122 20342 raft_consensus.cc:348] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134a6e21fa7543ae9c3b7099210a86b7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40777 } } peers { permanent_uuid: "0107b7cf7e214164847752340701bd3d" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33339 } } peers { permanent_uuid: "7b468b468cee42338474bb5bf3055254" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34389 } }
I20260928 03:07:36.889631 20342 raft_consensus.cc:374] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:07:36.889860 20342 raft_consensus.cc:729] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0107b7cf7e214164847752340701bd3d, State: Initialized, Role: FOLLOWER
I20260928 03:07:36.889829 20341 raft_consensus.cc:348] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134a6e21fa7543ae9c3b7099210a86b7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40777 } } peers { permanent_uuid: "0107b7cf7e214164847752340701bd3d" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33339 } } peers { permanent_uuid: "7b468b468cee42338474bb5bf3055254" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34389 } }
I20260928 03:07:36.890280 20341 raft_consensus.cc:374] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:07:36.890110 20343 consensus_queue.cc:260] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134a6e21fa7543ae9c3b7099210a86b7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40777 } } peers { permanent_uuid: "0107b7cf7e214164847752340701bd3d" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33339 } } peers { permanent_uuid: "7b468b468cee42338474bb5bf3055254" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34389 } }
I20260928 03:07:36.890381 20341 raft_consensus.cc:729] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b468b468cee42338474bb5bf3055254, State: Initialized, Role: FOLLOWER
I20260928 03:07:36.890489 20342 consensus_queue.cc:260] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134a6e21fa7543ae9c3b7099210a86b7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40777 } } peers { permanent_uuid: "0107b7cf7e214164847752340701bd3d" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33339 } } peers { permanent_uuid: "7b468b468cee42338474bb5bf3055254" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34389 } }
I20260928 03:07:36.891033 20341 consensus_queue.cc:260] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134a6e21fa7543ae9c3b7099210a86b7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40777 } } peers { permanent_uuid: "0107b7cf7e214164847752340701bd3d" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33339 } } peers { permanent_uuid: "7b468b468cee42338474bb5bf3055254" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34389 } }
I20260928 03:07:36.891793 20313 heartbeater.cc:503] Master 127.6.192.62:41331 was elected leader, sending a full tablet report...
I20260928 03:07:36.892525 20343 ts_tablet_manager.cc:1428] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260928 03:07:36.895018 20342 ts_tablet_manager.cc:1428] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260928 03:07:36.895359 20341 ts_tablet_manager.cc:1428] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260928 03:07:36.895907 20034 heartbeater.cc:503] Master 127.6.192.62:41331 was elected leader, sending a full tablet report...
W20260928 03:07:36.963570 20314 tablet.cc:2369] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:07:37.060257 20349 raft_consensus.cc:482] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:07:37.060635 20349 raft_consensus.cc:504] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134a6e21fa7543ae9c3b7099210a86b7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40777 } } peers { permanent_uuid: "0107b7cf7e214164847752340701bd3d" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33339 } } peers { permanent_uuid: "7b468b468cee42338474bb5bf3055254" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34389 } }
I20260928 03:07:37.062512 20349 leader_election.cc:288] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 134a6e21fa7543ae9c3b7099210a86b7 (127.6.192.3:40777), 0107b7cf7e214164847752340701bd3d (127.6.192.2:33339)
I20260928 03:07:37.072381 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:37.062961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:48371 (local address 127.6.192.3:40777)
0928 03:07:37.063167 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:37.063174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:37.064117 (+ 943us) server_negotiation.cc:407] Connection header received
0928 03:07:37.064233 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:37.064238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:37.064312 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:37.064401 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:07:37.065306 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:37.066402 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:37.069596 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:37.070962 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:37.071074 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:37.071164 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:37.071972 (+ 808us) server_negotiation.cc:299] Negotiation successful
0928 03:07:37.072116 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 03:07:37.072383 20338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:37.062976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:60489 (local address 127.6.192.2:33339)
0928 03:07:37.063169 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:37.063176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:37.063932 (+ 756us) server_negotiation.cc:407] Connection header received
0928 03:07:37.064158 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:37.064164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:37.064249 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:37.064391 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:07:37.065306 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:37.066402 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:37.069596 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:37.070882 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:37.071033 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:37.071146 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:37.071973 (+ 827us) server_negotiation.cc:299] Negotiation successful
0928 03:07:37.072116 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 03:07:37.072535 20351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:37.062968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:40777 (local address 127.6.192.1:48371)
0928 03:07:37.063992 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:37.064022 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:37.064133 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:37.064432 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:37.064437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:37.064527 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:07:37.065114 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:37.065122 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:37.066556 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:37.066561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:37.069327 (+ 2766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:37.069346 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:37.071324 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:37.071329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:37.071688 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:37.071726 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 03:07:37.071845 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 03:07:37.072006 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":275,"spinlock_wait_cycles":24960,"thread_start_us":662,"threads_started":1}
I20260928 03:07:37.072897 20350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:37.062854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:33339 (local address 127.6.192.1:60489)
0928 03:07:37.063783 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:37.063807 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:37.063970 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:37.064432 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:37.064437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:37.064462 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:07:37.065074 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:37.065093 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:37.066555 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:37.066561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:37.069332 (+ 2771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:37.069347 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:37.071301 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:37.071314 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:37.071802 (+ 488us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:37.071844 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 03:07:37.071948 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 03:07:37.072057 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":302,"thread_start_us":495,"threads_started":1}
I20260928 03:07:37.073540 20268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df5c26c63114fa7bb8fea4cefbc2184" candidate_uuid: "7b468b468cee42338474bb5bf3055254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "134a6e21fa7543ae9c3b7099210a86b7" is_pre_election: true
I20260928 03:07:37.074036 20268 raft_consensus.cc:2454] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b468b468cee42338474bb5bf3055254 in term 0.
I20260928 03:07:37.074048 20134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df5c26c63114fa7bb8fea4cefbc2184" candidate_uuid: "7b468b468cee42338474bb5bf3055254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0107b7cf7e214164847752340701bd3d" is_pre_election: true
I20260928 03:07:37.074623 20134 raft_consensus.cc:2454] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b468b468cee42338474bb5bf3055254 in term 0.
I20260928 03:07:37.074728 19925 leader_election.cc:302] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 134a6e21fa7543ae9c3b7099210a86b7, 7b468b468cee42338474bb5bf3055254; no voters:
I20260928 03:07:37.075181 20349 raft_consensus.cc:2781] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:07:37.075345 20349 raft_consensus.cc:482] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:07:37.075488 20349 raft_consensus.cc:3037] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:07:37.077744 20349 raft_consensus.cc:504] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134a6e21fa7543ae9c3b7099210a86b7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40777 } } peers { permanent_uuid: "0107b7cf7e214164847752340701bd3d" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33339 } } peers { permanent_uuid: "7b468b468cee42338474bb5bf3055254" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34389 } }
I20260928 03:07:37.078423 20349 leader_election.cc:288] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [CANDIDATE]: Term 1 election: Requested vote from peers 134a6e21fa7543ae9c3b7099210a86b7 (127.6.192.3:40777), 0107b7cf7e214164847752340701bd3d (127.6.192.2:33339)
I20260928 03:07:37.079048 20268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df5c26c63114fa7bb8fea4cefbc2184" candidate_uuid: "7b468b468cee42338474bb5bf3055254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "134a6e21fa7543ae9c3b7099210a86b7"
I20260928 03:07:37.079304 20268 raft_consensus.cc:3037] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:07:37.079303 20134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df5c26c63114fa7bb8fea4cefbc2184" candidate_uuid: "7b468b468cee42338474bb5bf3055254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0107b7cf7e214164847752340701bd3d"
I20260928 03:07:37.079573 20134 raft_consensus.cc:3037] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:07:37.081626 20268 raft_consensus.cc:2454] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b468b468cee42338474bb5bf3055254 in term 1.
W20260928 03:07:37.082044 20180 tablet.cc:2369] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:07:37.082111 19925 leader_election.cc:302] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 134a6e21fa7543ae9c3b7099210a86b7, 7b468b468cee42338474bb5bf3055254; no voters:
I20260928 03:07:37.082465 20349 raft_consensus.cc:2781] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:07:37.082489 20134 raft_consensus.cc:2454] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b468b468cee42338474bb5bf3055254 in term 1.
I20260928 03:07:37.083112 20349 raft_consensus.cc:686] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [term 1 LEADER]: Becoming Leader. State: Replica: 7b468b468cee42338474bb5bf3055254, State: Running, Role: LEADER
I20260928 03:07:37.083712 20349 consensus_queue.cc:237] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134a6e21fa7543ae9c3b7099210a86b7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40777 } } peers { permanent_uuid: "0107b7cf7e214164847752340701bd3d" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33339 } } peers { permanent_uuid: "7b468b468cee42338474bb5bf3055254" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34389 } }
I20260928 03:07:37.088934 19835 catalog_manager.cc:5495] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b468b468cee42338474bb5bf3055254 (127.6.192.1). New cstate: current_term: 1 leader_uuid: "7b468b468cee42338474bb5bf3055254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134a6e21fa7543ae9c3b7099210a86b7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0107b7cf7e214164847752340701bd3d" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b468b468cee42338474bb5bf3055254" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 34389 } health_report { overall_health: HEALTHY } } }
W20260928 03:07:37.115005 20035 tablet.cc:2369] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:07:37.131795 20339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:37.126220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39724 (local address 127.6.192.1:34389)
0928 03:07:37.126438 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:37.126443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:37.126483 (+ 40us) server_negotiation.cc:407] Connection header received
0928 03:07:37.126571 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:37.126575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:37.126651 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:37.126797 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:07:37.127095 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:37.127886 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:37.129392 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:37.129883 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:37.129983 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:37.130589 (+ 606us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:07:37.131111 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:07:37.131244 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:37.131354 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 03:07:37.131491 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 03:07:37.141204 20134 raft_consensus.cc:1261] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d [term 1 FOLLOWER]: Refusing update from remote peer 7b468b468cee42338474bb5bf3055254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:07:37.141201 20268 raft_consensus.cc:1261] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7 [term 1 FOLLOWER]: Refusing update from remote peer 7b468b468cee42338474bb5bf3055254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:07:37.142522 20352 consensus_queue.cc:1035] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0107b7cf7e214164847752340701bd3d" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:07:37.142908 20349 consensus_queue.cc:1035] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "134a6e21fa7543ae9c3b7099210a86b7" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 40777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:07:37.155064 20357 mvcc.cc:205] Tried to move back new op lower bound from 7334153654838431744 to 7334153654619410432. Current Snapshot: MvccSnapshot[applied={T|T < 7334153654838431744}]
I20260928 03:07:37.168814 19835 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:07:37.169219 19835 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:07:37.173025 19835 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 809ac6bd2f8840b48323c598fdd9e37c: Sending DeleteTablet for 3 replicas of tablet 9df5c26c63114fa7bb8fea4cefbc2184
I20260928 03:07:37.174328 20114 tablet_service.cc:1514] Processing DeleteTablet for tablet 9df5c26c63114fa7bb8fea4cefbc2184 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:07:37 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:52070
I20260928 03:07:37.174352 20248 tablet_service.cc:1514] Processing DeleteTablet for tablet 9df5c26c63114fa7bb8fea4cefbc2184 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:07:37 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:41596
I20260928 03:07:37.175201 19969 tablet_service.cc:1514] Processing DeleteTablet for tablet 9df5c26c63114fa7bb8fea4cefbc2184 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:07:37 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:39720
I20260928 03:07:37.180110 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 19874
I20260928 03:07:37.186638 20365 tablet_replica.cc:330] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d: stopping tablet replica
I20260928 03:07:37.187129 20365 raft_consensus.cc:2229] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:07:37.189836 20365 raft_consensus.cc:2258] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:07:37.190392 20366 tablet_replica.cc:330] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254: stopping tablet replica
I20260928 03:07:37.190851 20366 raft_consensus.cc:2229] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:07:37.191004 20364 tablet_replica.cc:330] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7: stopping tablet replica
I20260928 03:07:37.191462 20364 raft_consensus.cc:2229] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:07:37.191718 20366 raft_consensus.cc:2258] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:07:37.191982 20364 raft_consensus.cc:2258] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:07:37.193766 20365 ts_tablet_manager.cc:1905] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:07:37.194187 20364 ts_tablet_manager.cc:1905] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:07:37.201167 20365 ts_tablet_manager.cc:1918] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:07:37.201413 20365 log.cc:1196] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/9df5c26c63114fa7bb8fea4cefbc2184
I20260928 03:07:37.201790 20364 ts_tablet_manager.cc:1918] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:07:37.202015 20365 ts_tablet_manager.cc:1939] T 9df5c26c63114fa7bb8fea4cefbc2184 P 0107b7cf7e214164847752340701bd3d: Deleting consensus metadata
I20260928 03:07:37.202083 20364 log.cc:1196] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/9df5c26c63114fa7bb8fea4cefbc2184
I20260928 03:07:37.202724 20364 ts_tablet_manager.cc:1939] T 9df5c26c63114fa7bb8fea4cefbc2184 P 134a6e21fa7543ae9c3b7099210a86b7: Deleting consensus metadata
I20260928 03:07:37.204074 19823 catalog_manager.cc:4841] TS 0107b7cf7e214164847752340701bd3d (127.6.192.2:33339): tablet 9df5c26c63114fa7bb8fea4cefbc2184 (table test-table [id=764a3104ff11440989f0eefc0676c673]) successfully deleted
I20260928 03:07:37.204907 19824 catalog_manager.cc:4841] TS 134a6e21fa7543ae9c3b7099210a86b7 (127.6.192.3:40777): tablet 9df5c26c63114fa7bb8fea4cefbc2184 (table test-table [id=764a3104ff11440989f0eefc0676c673]) successfully deleted
I20260928 03:07:37.205559 20366 ts_tablet_manager.cc:1905] T 9df5c26c63114fa7bb8fea4cefbc2184 P 7b468b468cee42338474bb5bf3055254: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:07:37.207428 20339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:37.181665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.6.192.1:34389)
0928 03:07:37.181908 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:37.181917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:37.181939 (+ 22us) server_negotiation.cc:407] Connection header received
0928 03:07:37.182021 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:37.182029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:37.182134 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:37.182253 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:37.188599 (+ 6346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:37.189987 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:37.192195 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:37.192901 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:37.193033 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:37.197148 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:37.197163 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:37.197184 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:37.197230 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:37.201917 (+ 4687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:37.204298 (+ 2381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:37.204309 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:37.204313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:37.204496 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:37.205034 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:37.205040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:37.205044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:37.205590 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:37.205914 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:37.206548 (+ 634us) server_negotiation.cc:299] Negotiation successful
0928 03:07:37.206840 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 03:07:37.208492 20029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:37.308339 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 19874
W20260928 03:07:37.325829 19824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.192.1:34389 (error 108)
I20260928 03:07:37.326711 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 20038
I20260928 03:07:37.327615 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:37.327061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:34389 (local address 127.0.0.1:39740)
0928 03:07:37.327285 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:37.327421 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:34389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260928 03:07:37.327925 19824 catalog_manager.cc:4584] TS 7b468b468cee42338474bb5bf3055254 (127.6.192.1:34389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9df5c26c63114fa7bb8fea4cefbc2184: Network error: Client connection negotiation failed: client connection to 127.6.192.1:34389: connect: Connection refused (error 111)
I20260928 03:07:37.341113 20338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:37.327633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52074 (local address 127.6.192.2:33339)
0928 03:07:37.327895 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:37.327901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:37.327928 (+ 27us) server_negotiation.cc:407] Connection header received
0928 03:07:37.328000 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:37.328005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:37.328092 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:37.328212 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:37.329621 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:37.330404 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:37.331808 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:37.332231 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:37.332376 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:37.335032 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:37.335051 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:37.335067 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:37.335105 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:37.338081 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:37.338875 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:37.338883 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:37.338885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:37.339031 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:37.339467 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:37.339474 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:37.339484 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:37.339835 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:37.340181 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:37.340504 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 03:07:37.340733 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 03:07:37.341905 20174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:37.381552 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:37.381026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:34389 (local address 127.0.0.1:39752)
0928 03:07:37.381262 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:37.381404 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:34389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 03:07:37.382728 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:37.382359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:34389 (local address 127.0.0.1:39762)
0928 03:07:37.382511 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:37.382606 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:34389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 03:07:37.433267 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 20038
I20260928 03:07:37.443336 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:37.442860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:34389 (local address 127.0.0.1:39770)
0928 03:07:37.443071 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:37.443163 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:34389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 03:07:37.444517 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:37.444179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:34389 (local address 127.0.0.1:39784)
0928 03:07:37.444325 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:37.444405 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:34389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 03:07:37.449795 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 20183
I20260928 03:07:37.464367 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:37.450717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.6.192.3:40777)
0928 03:07:37.450949 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:37.450955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:37.450976 (+ 21us) server_negotiation.cc:407] Connection header received
0928 03:07:37.451178 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:37.451183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:37.451256 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:37.451367 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:37.452760 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:37.453493 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:37.454831 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:37.455250 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:37.455398 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:37.458171 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:37.458187 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:37.458203 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:37.458239 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:37.461251 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:37.462221 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:37.462229 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:37.462232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:37.462406 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:37.462809 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:37.462815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:37.462818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:37.463179 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:37.463502 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:37.463837 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 03:07:37.464016 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 03:07:37.465126 20308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:37.525241 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:37.524617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:34389 (local address 127.0.0.1:39794)
0928 03:07:37.524888 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:37.525010 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:34389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 03:07:37.526443 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:37.525996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:34389 (local address 127.0.0.1:39804)
0928 03:07:37.526206 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:37.526305 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:34389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 03:07:37.556092 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 20183
I20260928 03:07:37.572724 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 19804
I20260928 03:07:37.574118 19867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:37.686780 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 19804
I20260928 03:07:37.711602 6912 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 03:07:38.047782 6912 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 03:07:38.047971 6912 mini_ranger.cc:66] Stopping Ranger...
I20260928 03:07:38.394796 6912 mini_ranger.cc:68] Stopped Ranger
2026-09-28 03:07:38.395 UTC [18854] LOG: received smart shutdown request
2026-09-28 03:07:38.398 UTC [18854] LOG: background worker "logical replication launcher" (PID 18863) exited with exit code 1
2026-09-28 03:07:38.399 UTC [18858] LOG: shutting down
2026-09-28 03:07:38.602 UTC [18854] LOG: database system is shut down
2026-09-28T03:07:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (58089 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:07:38 dist-test-slave-fjww krb5kdc[20380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:07:38 dist-test-slave-fjww krb5kdc[20380](info): set up 2 sockets
Sep 28 03:07:38 dist-test-slave-fjww krb5kdc[20380](info): commencing operation
krb5kdc: starting...
W20260928 03:07:39.848385 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:07:40 dist-test-slave-fjww krb5kdc[20380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:07:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:07:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:07:40.104723 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:34575
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:34575,127.6.192.61:33571,127.6.192.60:40461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:07:40.448012 20393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:40.448319 20393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:40.448503 20393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:40.456028 20393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:07:40.456157 20393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:40.456270 20393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:40.456389 20393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:07:40.456475 20393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:07:40.466657 20393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:34575,127.6.192.61:33571,127.6.192.60:40461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:34575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:40.468142 20393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:40.469532 20393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:40.483108 20398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:40.483053 20401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:40.483057 20399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:40.484212 20393 server_base.cc:1034] running on GCE node
I20260928 03:07:40.485150 20393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:40.486665 20393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:40.487964 20393 hybrid_clock.cc:648] HybridClock initialized: now 1790564860487913 us; error 66 us; skew 500 ppm
I20260928 03:07:40.488421 20393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:40 dist-test-slave-fjww krb5kdc[20380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564860, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:40.497663 20393 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:07:40.499571 20393 webserver.cc:460] Webserver started at http://127.6.192.62:37681/ using document root <none> and password file <none>
I20260928 03:07:40.500360 20393 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:40.500515 20393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:40.500857 20393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:07:40.504455 20393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "ac638581e2804945bf1025eca0036401"
format_stamp: "Formatted at 2026-09-28 03:07:40 on dist-test-slave-fjww"
server_key: "cf18ff3aaa09042c142071fb17ea3384"
server_key_iv: "f3235221c8a9e4b2a1976b252c04937e"
server_key_version: "encryptionkey@0"
I20260928 03:07:40.505513 20393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "ac638581e2804945bf1025eca0036401"
format_stamp: "Formatted at 2026-09-28 03:07:40 on dist-test-slave-fjww"
server_key: "cf18ff3aaa09042c142071fb17ea3384"
server_key_iv: "f3235221c8a9e4b2a1976b252c04937e"
server_key_version: "encryptionkey@0"
I20260928 03:07:40.511102 20393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 03:07:40.515049 20408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:40.516640 20393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:07:40.516870 20393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "ac638581e2804945bf1025eca0036401"
format_stamp: "Formatted at 2026-09-28 03:07:40 on dist-test-slave-fjww"
server_key: "cf18ff3aaa09042c142071fb17ea3384"
server_key_iv: "f3235221c8a9e4b2a1976b252c04937e"
server_key_version: "encryptionkey@0"
I20260928 03:07:40.517105 20393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:40.538390 20393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:40.543275 20393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:40.543612 20393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:40.563163 20393 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:34575
I20260928 03:07:40.563194 20459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:34575 every 8 connection(s)
I20260928 03:07:40.564654 20393 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:07:40.568593 20460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:07:40.570346 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 20393
I20260928 03:07:40.570598 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:07:40.571517 6912 external_mini_cluster.cc:1428] Setting key e532d51080232e063e0a5bd13dc019ae
I20260928 03:07:40.579026 20460 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.192.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.61@KRBTEST.COM" created.
Sep 28 03:07:40 dist-test-slave-fjww krb5kdc[20380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564860, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:07:40.608824 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:40.584747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49956 (local address 127.6.192.62:34575)
0928 03:07:40.585511 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:40.585527 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:40.585595 (+ 68us) server_negotiation.cc:407] Connection header received
0928 03:07:40.586922 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:40.586929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:40.587247 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:40.587490 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:40.588894 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:40.590165 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:40.591743 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:40.592330 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:40.592551 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:40.599422 (+ 6871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:40.599490 (+ 68us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:40.599516 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:40.599570 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:40.604005 (+ 4435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:40.604821 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:40.604834 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:40.604840 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:40.605076 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:40.605493 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:40.605500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:40.605506 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:40.606283 (+ 777us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:40.606451 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:40.607206 (+ 755us) server_negotiation.cc:299] Negotiation successful
0928 03:07:40.607594 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":359,"threads_started":1}
I20260928 03:07:40.608850 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:40.583183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34575 (local address 127.0.0.1:49956)
0928 03:07:40.584133 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:40.584185 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:40.586476 (+ 2291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:40.587507 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:40.587514 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:40.588277 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:40.588679 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:40.588702 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:40.590350 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:40.590355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:40.591596 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:40.591609 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:40.592592 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:40.592599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:40.592809 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:40.593668 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:40.593696 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:40.599009 (+ 5313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:40.604193 (+ 5184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:40.604206 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:40.604217 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:40.604647 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:40.605196 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:40.605200 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:40.605202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:40.605370 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:40.606444 (+ 1074us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:40.606454 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:40.606732 (+ 278us) client_negotiation.cc:769] Sending connection context
0928 03:07:40.607046 (+ 314us) client_negotiation.cc:240] Negotiation successful
0928 03:07:40.607768 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":114,"thread_start_us":428,"threads_started":1}
I20260928 03:07:40.612249 20460 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:07:40.613497 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:40.613077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33571 (local address 127.0.0.1:60330)
0928 03:07:40.613245 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:40.613379 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 03:07:40.613919 20412 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33571: connect: Connection refused (error 111)
I20260928 03:07:40.615603 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:40.615188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33571 (local address 127.0.0.1:60346)
0928 03:07:40.615348 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:40.615426 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 03:07:40.615934 20460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.61:33571: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33571: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.61.keytab.
Entry for principal kudu/127.6.192.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.61.keytab.
I20260928 03:07:40.630774 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.61:33571
--webserver_interface=127.6.192.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.61.keytab
--principal=kudu/127.6.192.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:34575,127.6.192.61:33571,127.6.192.60:40461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:07:40.665292 20460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } attempt: 1
I20260928 03:07:40.666599 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:40.666079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33571 (local address 127.0.0.1:60348)
0928 03:07:40.666335 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:40.666448 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 03:07:40.667703 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:40.667310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33571 (local address 127.0.0.1:60364)
0928 03:07:40.667475 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:40.667564 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 03:07:40.667898 20460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.61:33571: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33571: connect: Connection refused (error 111)
I20260928 03:07:40.736148 20460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } attempt: 2
I20260928 03:07:40.737403 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:40.736898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33571 (local address 127.0.0.1:60368)
0928 03:07:40.737161 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:40.737253 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 03:07:40.738480 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:40.738114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33571 (local address 127.0.0.1:60380)
0928 03:07:40.738260 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:40.738347 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 03:07:40.738667 20460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.61:33571: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33571: connect: Connection refused (error 111)
I20260928 03:07:40.829963 20460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } attempt: 3
I20260928 03:07:40.831336 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:40.830823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33571 (local address 127.0.0.1:60394)
0928 03:07:40.831065 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:40.831191 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 03:07:40.832501 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:40.832103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33571 (local address 127.0.0.1:60404)
0928 03:07:40.832260 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:40.832346 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 03:07:40.832706 20460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.61:33571: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33571: connect: Connection refused (error 111)
I20260928 03:07:40.975960 20460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } attempt: 4
I20260928 03:07:40.977205 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:40.976626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33571 (local address 127.0.0.1:60408)
0928 03:07:40.976943 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:40.977053 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260928 03:07:40.978230 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:40.977867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33571 (local address 127.0.0.1:60410)
0928 03:07:40.978015 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:40.978110 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 03:07:40.978420 20460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.61:33571: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33571: connect: Connection refused (error 111)
W20260928 03:07:40.987763 20467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:40.988122 20467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:40.988284 20467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:40.996035 20467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:07:40.996196 20467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:40.996308 20467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:40.996414 20467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:07:40.996501 20467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:07:41.006975 20467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:34575,127.6.192.61:33571,127.6.192.60:40461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.61.keytab
--principal=kudu/127.6.192.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.61:33571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.192.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:41.008553 20467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:41.010067 20467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:41.023741 20472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:41.023787 20475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:41.023787 20473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:41.025272 20467 server_base.cc:1034] running on GCE node
I20260928 03:07:41.026134 20467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:41.027838 20467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:41.029112 20467 hybrid_clock.cc:648] HybridClock initialized: now 1790564861029071 us; error 45 us; skew 500 ppm
I20260928 03:07:41.029680 20467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:41 dist-test-slave-fjww krb5kdc[20380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564861, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:41.033952 20467 init.cc:377] Logged in from keytab as kudu/127.6.192.61@KRBTEST.COM (short username kudu)
I20260928 03:07:41.035758 20467 webserver.cc:460] Webserver started at http://127.6.192.61:42821/ using document root <none> and password file <none>
I20260928 03:07:41.036566 20467 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:41.036731 20467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:41.037082 20467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:07:41.040625 20467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-1/data/instance:
uuid: "baf0bb93100347cca9585267eb0b37a1"
format_stamp: "Formatted at 2026-09-28 03:07:41 on dist-test-slave-fjww"
server_key: "bb2f5d44df9d4185ec0d8d4ec0b6b99c"
server_key_iv: "00932e7bf33c99738c1d0abbb963dbb2"
server_key_version: "encryptionkey@0"
I20260928 03:07:41.041579 20467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-1/wal/instance:
uuid: "baf0bb93100347cca9585267eb0b37a1"
format_stamp: "Formatted at 2026-09-28 03:07:41 on dist-test-slave-fjww"
server_key: "bb2f5d44df9d4185ec0d8d4ec0b6b99c"
server_key_iv: "00932e7bf33c99738c1d0abbb963dbb2"
server_key_version: "encryptionkey@0"
I20260928 03:07:41.047889 20467 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 03:07:41.051889 20482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:41.053521 20467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:07:41.053742 20467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-1/wal
uuid: "baf0bb93100347cca9585267eb0b37a1"
format_stamp: "Formatted at 2026-09-28 03:07:41 on dist-test-slave-fjww"
server_key: "bb2f5d44df9d4185ec0d8d4ec0b6b99c"
server_key_iv: "00932e7bf33c99738c1d0abbb963dbb2"
server_key_version: "encryptionkey@0"
I20260928 03:07:41.053985 20467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:41.077522 20467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:41.082113 20467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:41.082388 20467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:41.102121 20467 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.61:33571
I20260928 03:07:41.102165 20533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.61:33571 every 8 connection(s)
I20260928 03:07:41.103778 20467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-1/data/info.pb
I20260928 03:07:41.105309 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 20467
I20260928 03:07:41.105541 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-1/wal/instance
I20260928 03:07:41.106081 6912 external_mini_cluster.cc:1428] Setting key 9105776ef5b76bafc627a764ea9c93b6
I20260928 03:07:41.108959 20534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:07:41.118003 20534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.192.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.60@KRBTEST.COM" created.
Sep 28 03:07:41 dist-test-slave-fjww krb5kdc[20380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564861, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.61@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:07:41.143585 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.120954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49972 (local address 127.6.192.62:34575)
0928 03:07:41.121494 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:41.121499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:41.122785 (+ 1286us) server_negotiation.cc:407] Connection header received
0928 03:07:41.124880 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:41.124886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:41.125011 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:41.125148 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:41.127665 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.128499 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.130746 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.131231 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.131404 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.135910 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:41.135921 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:41.135929 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:41.136006 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:41.139953 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.140870 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.140876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.140878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.141013 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.141524 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.141528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.141531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.141838 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:41.142078 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:41.143026 (+ 948us) server_negotiation.cc:299] Negotiation successful
0928 03:07:41.143215 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":312,"threads_started":1}
I20260928 03:07:41.145136 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.121703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34575 (local address 127.0.0.1:49972)
0928 03:07:41.122624 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.122666 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:41.124562 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:41.125510 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:41.125525 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:41.126424 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:41.127393 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:41.127426 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.128700 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.128706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:41.130532 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:41.130579 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.131589 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.131595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:41.131923 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.132677 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:41.132702 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:41.135618 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:41.140160 (+ 4542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:41.140169 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:41.140184 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:41.140694 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:41.141141 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:41.141147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:41.141150 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:41.141349 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:41.142035 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:41.142044 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:41.142587 (+ 543us) client_negotiation.cc:769] Sending connection context
0928 03:07:41.142913 (+ 326us) client_negotiation.cc:240] Negotiation successful
0928 03:07:41.143553 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":340,"threads_started":1}
I20260928 03:07:41.147173 20534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:07:41 dist-test-slave-fjww krb5kdc[20380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564861, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.61@KRBTEST.COM for kudu/127.6.192.61@KRBTEST.COM
I20260928 03:07:41.165370 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.148151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33571 (local address 127.0.0.1:60418)
0928 03:07:41.148336 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.148354 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:41.148449 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:41.149645 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:41.149649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:41.150135 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:41.150271 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:41.150280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.151797 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.151807 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:41.152775 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:41.152788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.153652 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.153657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:41.153938 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.154571 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:41.154589 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:41.159341 (+ 4752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:41.162671 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:41.162677 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:41.162681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:41.163073 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:41.163514 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:41.163520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:41.163524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:41.163666 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:41.164304 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:41.164309 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:41.164458 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 03:07:41.164613 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 03:07:41.164930 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":45}
I20260928 03:07:41.165429 20540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.148487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60418 (local address 127.6.192.61:33571)
0928 03:07:41.149163 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:41.149169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:41.149196 (+ 27us) server_negotiation.cc:407] Connection header received
0928 03:07:41.149245 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:41.149251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:41.149485 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:41.149657 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:41.150437 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.151605 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.152923 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.153495 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.153639 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.159617 (+ 5978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:41.159631 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:41.159645 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:41.159684 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:41.162491 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.163225 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.163236 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.163240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.163370 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.163772 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.163780 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.163784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.164149 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:41.164403 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:41.164771 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 03:07:41.165043 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":522,"thread_start_us":387,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.60.keytab.
Entry for principal kudu/127.6.192.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.60.keytab.
I20260928 03:07:41.166978 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.60:40461
--webserver_interface=127.6.192.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.60.keytab
--principal=kudu/127.6.192.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:34575,127.6.192.61:33571,127.6.192.60:40461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:07:41.168656 20534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:07:41.170002 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.169389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:58522)
0928 03:07:41.169610 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.169769 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 03:07:41.170284 20484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
I20260928 03:07:41.172094 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.171611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:58534)
0928 03:07:41.171812 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.171929 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 03:07:41.172325 20534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:40461: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
I20260928 03:07:41.221657 20534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } attempt: 1
I20260928 03:07:41.222947 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.222438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:41744)
0928 03:07:41.222635 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.222749 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 03:07:41.224143 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.223714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:41746)
0928 03:07:41.223903 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.223995 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 03:07:41.224376 20534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:40461: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
I20260928 03:07:41.277729 20460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } attempt: 5
Sep 28 03:07:41 dist-test-slave-fjww krb5kdc[20380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564860, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for kudu/127.6.192.61@KRBTEST.COM
I20260928 03:07:41.292027 20540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.278614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43766 (local address 127.6.192.61:33571)
0928 03:07:41.278842 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:41.278849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:41.278871 (+ 22us) server_negotiation.cc:407] Connection header received
0928 03:07:41.279119 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:41.279124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:41.279186 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:41.279297 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:41.280122 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.280815 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.281989 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.282445 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.282587 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.286228 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:41.286234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:41.286238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:41.286275 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:41.289200 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.289957 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.289966 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.289970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.290106 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.290506 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.290512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.290515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.290822 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:41.290977 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:41.291412 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 03:07:41.291680 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260928 03:07:41.292043 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.278512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33571 (local address 127.0.0.1:43766)
0928 03:07:41.278702 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.278725 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:41.278921 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:41.279327 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:41.279331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:41.279790 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:41.279969 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:41.279977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.280954 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.280960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:41.281834 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:41.281856 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.282602 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.282608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:41.282793 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.283541 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:41.283560 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:41.285986 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:41.289392 (+ 3406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:41.289398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:41.289402 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:41.289817 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:41.290235 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:41.290241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:41.290244 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:41.290390 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:41.290954 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:41.290959 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:41.291145 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 03:07:41.291290 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 03:07:41.291670 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 03:07:41.292603 20534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } attempt: 2
I20260928 03:07:41.293501 20460 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:07:41.294397 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.293231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:41748)
0928 03:07:41.294073 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.294187 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 03:07:41.294793 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.294297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:41756)
0928 03:07:41.294501 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.294603 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 03:07:41.295629 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.295171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:41768)
0928 03:07:41.295353 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.295478 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 03:07:41.295948 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.295563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:41780)
0928 03:07:41.295730 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.295813 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 03:07:41.295957 20534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:40461: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
W20260928 03:07:41.296181 20460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:40461: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
I20260928 03:07:41.347491 20460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } attempt: 1
I20260928 03:07:41.348995 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.348314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:41790)
0928 03:07:41.348592 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.348745 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 03:07:41.350203 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.349817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:41806)
0928 03:07:41.349985 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.350073 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 03:07:41.350486 20460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:40461: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
I20260928 03:07:41.387228 20534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } attempt: 3
I20260928 03:07:41.388448 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.387987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:41808)
0928 03:07:41.388187 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.388280 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 03:07:41.389645 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.389202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:41824)
0928 03:07:41.389460 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.389531 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260928 03:07:41.389914 20534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:40461: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
I20260928 03:07:41.418748 20460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } attempt: 2
I20260928 03:07:41.420153 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.419565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:41840)
0928 03:07:41.419815 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.419945 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 03:07:41.421244 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.420923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:41846)
0928 03:07:41.421054 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.421125 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 03:07:41.421584 20460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:40461: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
W20260928 03:07:41.512426 20542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:41.512732 20542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:41.512969 20542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:41.520622 20542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:07:41.520799 20542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:41.520891 20542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:41.520992 20542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:07:41.521073 20542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:07:41.527824 20460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } attempt: 3
I20260928 03:07:41.529028 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.528574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:41858)
0928 03:07:41.528771 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.528862 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 03:07:41.530133 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.529750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:41874)
0928 03:07:41.529946 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.530023 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260928 03:07:41.530376 20460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:40461: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
I20260928 03:07:41.532339 20542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:34575,127.6.192.61:33571,127.6.192.60:40461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.60.keytab
--principal=kudu/127.6.192.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.60:40461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.192.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:41.533126 20534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } attempt: 4
I20260928 03:07:41.533934 20542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:41.534186 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.533723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:41888)
0928 03:07:41.533922 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.533999 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 03:07:41.535128 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.534792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:41904)
0928 03:07:41.534941 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.535028 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 03:07:41.535328 20534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:40461: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40461: connect: Connection refused (error 111)
I20260928 03:07:41.535598 20542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:41.547899 20547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:41.548329 20550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:41.548365 20548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:41.548907 20542 server_base.cc:1034] running on GCE node
I20260928 03:07:41.549780 20542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:41.551527 20542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:41.552901 20542 hybrid_clock.cc:648] HybridClock initialized: now 1790564861552831 us; error 60 us; skew 500 ppm
I20260928 03:07:41.553371 20542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:41 dist-test-slave-fjww krb5kdc[20380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564861, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:41.557178 20542 init.cc:377] Logged in from keytab as kudu/127.6.192.60@KRBTEST.COM (short username kudu)
I20260928 03:07:41.559092 20542 webserver.cc:460] Webserver started at http://127.6.192.60:37119/ using document root <none> and password file <none>
I20260928 03:07:41.560179 20542 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:41.560351 20542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:41.560685 20542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:07:41.564342 20542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-2/data/instance:
uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9"
format_stamp: "Formatted at 2026-09-28 03:07:41 on dist-test-slave-fjww"
server_key: "4a00894eafcda25f1d97bd9c966e141c"
server_key_iv: "38bf6d3a3ee8b279579aeab731891e6a"
server_key_version: "encryptionkey@0"
I20260928 03:07:41.565306 20542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-2/wal/instance:
uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9"
format_stamp: "Formatted at 2026-09-28 03:07:41 on dist-test-slave-fjww"
server_key: "4a00894eafcda25f1d97bd9c966e141c"
server_key_iv: "38bf6d3a3ee8b279579aeab731891e6a"
server_key_version: "encryptionkey@0"
I20260928 03:07:41.571292 20542 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 03:07:41.574904 20557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:41.576574 20542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:07:41.576790 20542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-2/wal
uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9"
format_stamp: "Formatted at 2026-09-28 03:07:41 on dist-test-slave-fjww"
server_key: "4a00894eafcda25f1d97bd9c966e141c"
server_key_iv: "38bf6d3a3ee8b279579aeab731891e6a"
server_key_version: "encryptionkey@0"
I20260928 03:07:41.576999 20542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:41.599781 20542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:41.604450 20542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:41.604796 20542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:41.624321 20542 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.60:40461
I20260928 03:07:41.624375 20608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.60:40461 every 8 connection(s)
I20260928 03:07:41.625744 20542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-2/data/info.pb
I20260928 03:07:41.629580 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:07:41.631853 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 20542
I20260928 03:07:41.632066 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790564644636890-6912-0/minicluster-data/master-2/wal/instance
I20260928 03:07:41.632547 6912 external_mini_cluster.cc:1428] Setting key 602aa36485e7887537bd97b6bc443e36
I20260928 03:07:41.638504 20609 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:07:41 dist-test-slave-fjww krb5kdc[20380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.60@KRBTEST.COM
Sep 28 03:07:41 dist-test-slave-fjww krb5kdc[20380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564861, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.60@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:07:41.657290 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.641604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35036 (local address 127.6.192.62:34575)
0928 03:07:41.641831 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:41.641840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:41.642549 (+ 709us) server_negotiation.cc:407] Connection header received
0928 03:07:41.642742 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:41.642749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:41.642852 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:41.642962 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:41.644083 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.644762 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.646302 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.647077 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.647228 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.650735 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:41.650743 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:41.650748 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:41.650787 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:41.653944 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.654875 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.654879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.654882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.655034 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.655409 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.655414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.655417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.655790 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:41.655945 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:41.656642 (+ 697us) server_negotiation.cc:299] Negotiation successful
0928 03:07:41.656934 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 03:07:41.657975 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.641565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34575 (local address 127.0.0.1:35036)
0928 03:07:41.642387 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.642439 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:41.642616 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:41.643011 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:41.643016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:41.643654 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:41.643920 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:41.643929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.644943 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.644950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:41.646155 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:41.646166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.647344 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.647349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:41.647544 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.648172 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:41.648194 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:41.650534 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:41.654106 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:41.654116 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:41.654121 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:41.654753 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:41.655143 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:41.655146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:41.655148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:41.655296 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:41.655944 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:41.655953 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:41.656199 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 03:07:41.656508 (+ 309us) client_negotiation.cc:240] Negotiation successful
0928 03:07:41.656819 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"spinlock_wait_cycles":22272,"thread_start_us":420,"threads_started":1}
I20260928 03:07:41.658015 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.634558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41916 (local address 127.6.192.60:40461)
0928 03:07:41.635460 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:41.635479 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:41.635541 (+ 62us) server_negotiation.cc:407] Connection header received
0928 03:07:41.637923 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:41.637952 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:41.638471 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:41.638968 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:41.640553 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.642443 (+ 1890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.644057 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.644683 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.644814 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.649503 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:41.649511 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:41.649524 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:41.649558 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:41.653272 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.654058 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.654069 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.654081 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.654258 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.654729 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.654732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.654734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.655539 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:41.655835 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:41.656264 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 03:07:41.656681 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":362,"threads_started":1}
I20260928 03:07:41.659798 20609 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:07:41 dist-test-slave-fjww krb5kdc[20380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564861, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.60@KRBTEST.COM for kudu/127.6.192.61@KRBTEST.COM
Sep 28 03:07:41 dist-test-slave-fjww krb5kdc[20380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:07:41.674497 20540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.660685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43770 (local address 127.6.192.61:33571)
0928 03:07:41.660888 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:41.660896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:41.660995 (+ 99us) server_negotiation.cc:407] Connection header received
0928 03:07:41.661197 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:41.661204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:41.661295 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:41.661417 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:41.662338 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.663021 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.664362 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.664814 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.664916 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.668436 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:41.668445 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:41.668450 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:41.668501 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:41.671310 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.672443 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.672447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.672449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.672591 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.673112 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.673118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.673123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.673449 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:41.673567 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:41.674043 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 03:07:41.674218 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 03:07:41.674635 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.660723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33571 (local address 127.0.0.1:43770)
0928 03:07:41.660904 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.660925 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:41.661041 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:41.661470 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:41.661476 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:41.661980 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:41.662193 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:41.662206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.663166 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.663172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:41.664196 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:41.664215 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.664972 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.664979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:41.665173 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.665795 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:41.665817 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:41.668187 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:41.671499 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:41.671506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:41.671510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:41.672299 (+ 789us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:41.672743 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:41.672748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:41.672751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:41.672973 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:41.673589 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:41.673594 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:41.673786 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 03:07:41.673939 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 03:07:41.674230 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 03:07:41.675907 20609 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:07:41.678077 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.661681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.6.192.62:34575)
0928 03:07:41.661835 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:41.661838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:41.661866 (+ 28us) server_negotiation.cc:407] Connection header received
0928 03:07:41.661919 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:41.661922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:41.661973 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:41.662048 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:41.663432 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.664171 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.665781 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.666189 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.666285 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.671132 (+ 4847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:41.671137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:41.671140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:41.671175 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:41.674869 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.675788 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.675795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.675798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.676020 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.676439 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.676443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.676445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.676826 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:41.677129 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:41.677439 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 03:07:41.677664 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 03:07:41 dist-test-slave-fjww krb5kdc[20380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564861, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.60@KRBTEST.COM for kudu/127.6.192.60@KRBTEST.COM
Sep 28 03:07:41 dist-test-slave-fjww krb5kdc[20380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.61@KRBTEST.COM
I20260928 03:07:41.691474 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.676624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:41924)
0928 03:07:41.676800 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.676819 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:41.676941 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:41.677262 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:41.677265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:41.677789 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:41.677966 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:41.677976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.679208 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.679213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:41.680495 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:41.680506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.681556 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.681561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:41.681731 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.682363 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:41.682386 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:41.684770 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:41.688611 (+ 3841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:41.688616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:41.688618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:41.689176 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:41.689586 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:41.689591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:41.689594 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:41.689721 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:41.690235 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:41.690240 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:41.690402 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 03:07:41.690562 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 03:07:41.691038 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":88}
I20260928 03:07:41.691811 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.676679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41924 (local address 127.6.192.60:40461)
0928 03:07:41.676830 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:41.676835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:41.676876 (+ 41us) server_negotiation.cc:407] Connection header received
0928 03:07:41.677054 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:41.677059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:41.677132 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:41.677234 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:41.678113 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.679067 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.680628 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.681381 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.681515 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.684966 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:41.684971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:41.684974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:41.685005 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:41.688452 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.689345 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.689349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.689352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.689463 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.689823 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.689831 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.689835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.690121 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:41.690272 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:41.690654 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 03:07:41.691231 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":211,"server-negotiator.queue_time_us":51}
I20260928 03:07:41.697018 20540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.679841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.6.192.61:33571)
0928 03:07:41.680022 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:41.680027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:41.680045 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:07:41.680121 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:41.680126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:41.680191 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:41.680283 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:41.681909 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.683124 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.684536 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.685127 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.685228 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.690494 (+ 5266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:41.690504 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:41.690512 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:41.690557 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:41.694134 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.695038 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.695042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.695044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.695179 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.695680 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.695684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.695686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.695958 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:41.696171 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:41.696538 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 03:07:41.696721 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 03:07:41.697937 20609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9: Bootstrap starting.
I20260928 03:07:41.701690 20609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9: Neither blocks nor log segments found. Creating new log.
I20260928 03:07:41.701936 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:07:41.702816 20609 log.cc:824] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9: Log is configured to *not* fsync() on all Append() calls
I20260928 03:07:41.705808 20609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9: No bootstrap required, opened a new log
I20260928 03:07:41.707595 20460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } attempt: 4
I20260928 03:07:41.726696 20609 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } } peers { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } } peers { permanent_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } }
I20260928 03:07:41.728055 20609 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:07:41.728263 20609 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0104c9e07dd40caa93ad4c5fce4d0d9, State: Initialized, Role: FOLLOWER
I20260928 03:07:41.728418 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.707743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35050 (local address 127.6.192.62:34575)
0928 03:07:41.709278 (+ 1535us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:41.709284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:41.709299 (+ 15us) server_negotiation.cc:407] Connection header received
0928 03:07:41.709346 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:41.709350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:41.709442 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:41.709528 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:41.711872 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.713093 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.715301 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.716098 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.716218 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.720316 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:41.720322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:41.720328 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:41.720385 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:41.724242 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.725234 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.725240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.725244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.725422 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.725883 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.725888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.725891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.726263 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:41.726544 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:41.727682 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0928 03:07:41.727993 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1417}
I20260928 03:07:41.729226 20609 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } } peers { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } } peers { permanent_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } }
Sep 28 03:07:41 dist-test-slave-fjww krb5kdc[20380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564860, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for kudu/127.6.192.60@KRBTEST.COM
I20260928 03:07:41.730340 20623 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } } peers { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } } peers { permanent_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } } }
I20260928 03:07:41.730819 20623 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:07:41.731913 20609 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:07:41.741981 20540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.707275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43790 (local address 127.6.192.61:33571)
0928 03:07:41.707514 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:41.707520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:41.708816 (+ 1296us) server_negotiation.cc:407] Connection header received
0928 03:07:41.709100 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:41.709106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:41.709190 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:41.709264 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:41.711883 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.713028 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.719030 (+ 6002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.719803 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.719932 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.731930 (+ 11998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:41.731938 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:41.731942 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:41.731993 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:41.736626 (+ 4633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.738012 (+ 1386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.738021 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.738028 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.738348 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.739036 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.739044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.739050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.739641 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:41.741216 (+ 1575us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:41.741392 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 03:07:41.741649 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 03:07:41.742022 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.708604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:41948)
0928 03:07:41.708844 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.708862 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:41.708957 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:41.714864 (+ 5907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:41.714870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:41.715488 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:41.715685 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:41.715703 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.724100 (+ 8397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.724109 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:41.725672 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:41.725686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.726822 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.726828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:41.727039 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.727719 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:41.727740 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:41.732003 (+ 4263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:41.738635 (+ 6632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:41.738643 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:41.738648 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:41.739263 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:41.739837 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:41.739841 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:41.739844 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:41.740047 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:41.740836 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:41.740842 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:41.741057 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 03:07:41.741217 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 03:07:41.741530 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260928 03:07:41.744091 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.708633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41948 (local address 127.6.192.60:40461)
0928 03:07:41.713841 (+ 5208us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:41.713847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:41.713881 (+ 34us) server_negotiation.cc:407] Connection header received
0928 03:07:41.713940 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:41.713946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:41.714041 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:41.717086 (+ 3045us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:41.717347 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.718787 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.725920 (+ 7133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.726632 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.731582 (+ 4950us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.732255 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:41.732261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:41.732265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:41.732302 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:41.738431 (+ 6129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.739483 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.739489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.739494 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.739672 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.740196 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.740201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.740204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.740679 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:41.742446 (+ 1767us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:41.742595 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 03:07:41.743662 (+ 1067us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5072,"thread_start_us":4829,"threads_started":1}
I20260928 03:07:41.746903 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.708426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41936 (local address 127.6.192.60:40461)
0928 03:07:41.712575 (+ 4149us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:41.712585 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:41.728117 (+ 15532us) server_negotiation.cc:407] Connection header received
0928 03:07:41.728510 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:41.728515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:41.728594 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:41.728670 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:41.730623 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.731765 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.733989 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.736357 (+ 2368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.736512 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.739882 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:41.739888 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:41.739895 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:41.739936 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:41.743727 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.744718 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.744724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.744729 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.744916 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.745343 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.745350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.745362 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.745642 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:41.745918 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:41.746311 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 03:07:41.746548 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3934}
I20260928 03:07:41.752122 20460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401: Bootstrap starting.
I20260928 03:07:41.756783 20460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401: Neither blocks nor log segments found. Creating new log.
I20260928 03:07:41.758275 20460 log.cc:824] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401: Log is configured to *not* fsync() on all Append() calls
I20260928 03:07:41.761449 20460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401: No bootstrap required, opened a new log
W20260928 03:07:41.762032 20634 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 03:07:41.762168 20634 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 03:07:41.757349 20542 master_runner.cc:186] Error getting master registration for 127.6.192.62:34575: OK, instance_id { permanent_uuid: "ac638581e2804945bf1025eca0036401" instance_seqno: 1790564860557519 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 03:07:41.768549 20542 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 03:07:41.770968 20460 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } } peers { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } } peers { permanent_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } }
I20260928 03:07:41.771301 20460 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:07:41.771484 20460 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac638581e2804945bf1025eca0036401, State: Initialized, Role: FOLLOWER
I20260928 03:07:41.772078 20460 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } } peers { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } } peers { permanent_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } }
I20260928 03:07:41.773048 20637 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } } peers { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } } peers { permanent_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } } }
I20260928 03:07:41.773442 20637 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:07:41.774375 20460 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 03:07:41.786386 20648 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 03:07:41.786633 20648 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 03:07:41.784340 20393 master_runner.cc:186] Error getting master registration for 127.6.192.61:33571: OK, instance_id { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" instance_seqno: 1790564861096453 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 03:07:41.794153 20393 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 03:07:41.796700 20414 tablet.cc:2369] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:07:41.834582 20534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } attempt: 5
Sep 28 03:07:41 dist-test-slave-fjww krb5kdc[20380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564861, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.61@KRBTEST.COM for kudu/127.6.192.60@KRBTEST.COM
I20260928 03:07:41.851760 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.835766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.6.192.60:40461)
0928 03:07:41.836022 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:41.836029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:41.836923 (+ 894us) server_negotiation.cc:407] Connection header received
0928 03:07:41.836999 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:41.837004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:41.837086 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:41.837201 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:41.838461 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.839607 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.840971 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.841510 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.841682 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.845564 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:41.845570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:41.845573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:41.845610 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:41.848733 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.849500 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.849507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.849511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.849659 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:41.850081 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:41.850087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:41.850091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:41.850429 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:41.850597 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:41.851048 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 03:07:41.851304 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 03:07:41.851775 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:41.836162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40461 (local address 127.0.0.1:41952)
0928 03:07:41.836630 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:41.836654 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:41.836793 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:41.837402 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:41.837407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:41.838029 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:41.838272 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:41.838287 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.839793 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.839798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:41.840801 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:41.840814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:41.841687 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:41.841697 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:41.841932 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:41.842639 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:41.842660 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:41.845299 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:41.848918 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:41.848925 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:41.848930 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:41.849362 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:41.849786 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:41.849793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:41.849797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:41.849958 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:41.850573 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:41.850578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:41.850754 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 03:07:41.850912 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 03:07:41.851261 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349}
W20260928 03:07:41.857589 20563 tablet.cc:2369] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:07:41.858809 20534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1: Bootstrap starting.
I20260928 03:07:41.862628 20534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1: Neither blocks nor log segments found. Creating new log.
I20260928 03:07:41.863958 20534 log.cc:824] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1: Log is configured to *not* fsync() on all Append() calls
I20260928 03:07:41.867013 20534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1: No bootstrap required, opened a new log
I20260928 03:07:41.876785 20534 raft_consensus.cc:348] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } } peers { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } } peers { permanent_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } }
I20260928 03:07:41.877319 20534 raft_consensus.cc:374] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:07:41.877535 20534 raft_consensus.cc:729] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf0bb93100347cca9585267eb0b37a1, State: Initialized, Role: FOLLOWER
I20260928 03:07:41.878265 20534 consensus_queue.cc:260] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } } peers { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } } peers { permanent_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } }
I20260928 03:07:41.879985 20651 sys_catalog.cc:455] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } } peers { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } } peers { permanent_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } } }
I20260928 03:07:41.880465 20651 sys_catalog.cc:458] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:07:41.881549 20534 sys_catalog.cc:564] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:07:41.900911 20623 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:07:41.901242 20623 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } } peers { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } } peers { permanent_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } }
I20260928 03:07:41.903075 20623 leader_election.cc:288] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac638581e2804945bf1025eca0036401 (127.6.192.62:34575), baf0bb93100347cca9585267eb0b37a1 (127.6.192.61:33571)
I20260928 03:07:41.904479 20435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac638581e2804945bf1025eca0036401" is_pre_election: true
I20260928 03:07:41.905128 20435 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0104c9e07dd40caa93ad4c5fce4d0d9 in term 0.
I20260928 03:07:41.906575 20559 leader_election.cc:302] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac638581e2804945bf1025eca0036401, e0104c9e07dd40caa93ad4c5fce4d0d9; no voters:
I20260928 03:07:41.906602 20509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf0bb93100347cca9585267eb0b37a1" is_pre_election: true
I20260928 03:07:41.907222 20509 raft_consensus.cc:2454] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0104c9e07dd40caa93ad4c5fce4d0d9 in term 0.
I20260928 03:07:41.907354 20623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:07:41.907527 20623 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:07:41.907644 20623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:07:41.911226 20623 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } } peers { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } } peers { permanent_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } }
I20260928 03:07:41.912586 20434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac638581e2804945bf1025eca0036401"
I20260928 03:07:41.912904 20434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:07:41.916379 20434 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0104c9e07dd40caa93ad4c5fce4d0d9 in term 1.
I20260928 03:07:41.916985 20559 leader_election.cc:302] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac638581e2804945bf1025eca0036401, e0104c9e07dd40caa93ad4c5fce4d0d9; no voters:
I20260928 03:07:41.920400 20623 leader_election.cc:288] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [CANDIDATE]: Term 1 election: Requested vote from peers ac638581e2804945bf1025eca0036401 (127.6.192.62:34575), baf0bb93100347cca9585267eb0b37a1 (127.6.192.61:33571)
I20260928 03:07:41.920536 20509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf0bb93100347cca9585267eb0b37a1"
I20260928 03:07:41.920822 20509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:07:41.920979 20623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:07:41.921121 20623 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [term 1 LEADER]: Becoming Leader. State: Replica: e0104c9e07dd40caa93ad4c5fce4d0d9, State: Running, Role: LEADER
I20260928 03:07:41.921950 20467 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260928 03:07:41.922217 20467 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 03:07:41.921664 20623 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } } peers { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } } peers { permanent_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } }
I20260928 03:07:41.924813 20509 raft_consensus.cc:2454] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0104c9e07dd40caa93ad4c5fce4d0d9 in term 1.
I20260928 03:07:41.937894 20664 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0104c9e07dd40caa93ad4c5fce4d0d9. Latest consensus state: current_term: 1 leader_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } } peers { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } } peers { permanent_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } } }
W20260928 03:07:41.938261 20662 catalog_manager.cc:1558] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 03:07:41.938416 20662 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 03:07:41.938504 20664 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [sys.catalog]: This master's current role is: LEADER
I20260928 03:07:41.949877 20666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:07:41.958750 20666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:07:41.976989 20509 raft_consensus.cc:1261] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1 [term 1 FOLLOWER]: Refusing update from remote peer e0104c9e07dd40caa93ad4c5fce4d0d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:07:41.979939 20664 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:07:41.989795 20434 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401 [term 1 FOLLOWER]: Refusing update from remote peer e0104c9e07dd40caa93ad4c5fce4d0d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:07:41.990973 20664 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:07:42.011277 20651 sys_catalog.cc:455] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0104c9e07dd40caa93ad4c5fce4d0d9. Latest consensus state: current_term: 1 leader_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } } peers { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } } peers { permanent_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } } }
I20260928 03:07:42.011644 20651 sys_catalog.cc:458] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:07:42.014958 20668 mvcc.cc:205] Tried to move back new op lower bound from 7334153674639421440 to 7334153674440298496. Current Snapshot: MvccSnapshot[applied={T|T < 7334153674639421440}]
I20260928 03:07:42.025398 20651 sys_catalog.cc:455] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } } peers { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } } peers { permanent_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } } }
I20260928 03:07:42.025708 20651 sys_catalog.cc:458] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:07:42.032332 20664 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } } peers { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } } peers { permanent_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } } }
I20260928 03:07:42.032692 20664 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [sys.catalog]: This master's current role is: LEADER
I20260928 03:07:42.032914 20637 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0104c9e07dd40caa93ad4c5fce4d0d9. Latest consensus state: current_term: 1 leader_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } } peers { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } } peers { permanent_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } } }
I20260928 03:07:42.033291 20637 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:07:42.035202 20670 mvcc.cc:205] Tried to move back new op lower bound from 7334153674639421440 to 7334153674440298496. Current Snapshot: MvccSnapshot[applied={T|T < 7334153674639421440}]
I20260928 03:07:42.042251 20664 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } } peers { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } } peers { permanent_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } } }
I20260928 03:07:42.042335 20637 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac638581e2804945bf1025eca0036401" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 34575 } } peers { permanent_uuid: "baf0bb93100347cca9585267eb0b37a1" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33571 } } peers { permanent_uuid: "e0104c9e07dd40caa93ad4c5fce4d0d9" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40461 } } }
I20260928 03:07:42.042562 20664 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 [sys.catalog]: This master's current role is: LEADER
I20260928 03:07:42.042613 20637 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:07:42.045565 20666 catalog_manager.cc:1347] Generated new cluster ID: 7fd78dcb8a014622bcdf4b5e7ede46de
I20260928 03:07:42.045686 20666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:07:42.121243 20666 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:07:42.127576 20666 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:07:42.162036 20666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9: Generated new TSK 0
I20260928 03:07:42.163241 20666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:07:42.788723 20648 catalog_manager.cc:1259] Loaded cluster ID: 7fd78dcb8a014622bcdf4b5e7ede46de
I20260928 03:07:42.788873 20648 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401: loading cluster ID for follower catalog manager: success
I20260928 03:07:42.791863 20648 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401: acquiring CA information for follower catalog manager: success
I20260928 03:07:42.793843 20648 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ac638581e2804945bf1025eca0036401: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 03:07:42.940650 20662 catalog_manager.cc:1259] Loaded cluster ID: 7fd78dcb8a014622bcdf4b5e7ede46de
I20260928 03:07:42.940754 20662 catalog_manager.cc:1552] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1: loading cluster ID for follower catalog manager: success
I20260928 03:07:42.943238 20662 catalog_manager.cc:1574] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1: acquiring CA information for follower catalog manager: success
I20260928 03:07:42.945782 20662 catalog_manager.cc:1602] T 00000000000000000000000000000000 P baf0bb93100347cca9585267eb0b37a1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 03:07:43.302304 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 20393
I20260928 03:07:43.303740 20455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:43.416505 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 20393
W20260928 03:07:43.437688 20559 connection.cc:535] client connection to 127.6.192.62:34575 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 03:07:43.437945 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 20467
W20260928 03:07:43.437994 20559 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 03:07:43.439188 20529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:43.440090 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:43.439210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:34575 (local address 127.0.0.1:35058)
0928 03:07:43.439699 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:43.439842 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":302,"threads_started":1}
W20260928 03:07:43.440383 20559 consensus_peers.cc:487] T 00000000000000000000000000000000 P e0104c9e07dd40caa93ad4c5fce4d0d9 -> Peer ac638581e2804945bf1025eca0036401 (127.6.192.62:34575): Couldn't send request to peer ac638581e2804945bf1025eca0036401. Status: Network error: Client connection negotiation failed: client connection to 127.6.192.62:34575: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 03:07:43.546001 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 20467
I20260928 03:07:43.566241 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 20542
I20260928 03:07:43.567536 20604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:43.678452 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 20542
2026-09-28T03:07:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4946 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:07:43 dist-test-slave-fjww krb5kdc[20686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:07:43 dist-test-slave-fjww krb5kdc[20686](info): set up 2 sockets
Sep 28 03:07:43 dist-test-slave-fjww krb5kdc[20686](info): commencing operation
krb5kdc: starting...
W20260928 03:07:44.828315 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:07:44 dist-test-slave-fjww krb5kdc[20686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:07:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:07:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:07:45.088002 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:42845
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:45741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:42845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:07:45.441779 20699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:45.442108 20699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:45.442204 20699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:45.449771 20699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:07:45.449901 20699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:45.449950 20699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:45.449999 20699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:07:45.450052 20699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:07:45.460656 20699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:45741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:42845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:42845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:45.462081 20699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:45.463470 20699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:45.477228 20704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:45.477459 20705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:45.477259 20707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:45.477888 20699 server_base.cc:1034] running on GCE node
I20260928 03:07:45.479151 20699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:45.480830 20699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:45.482116 20699 hybrid_clock.cc:648] HybridClock initialized: now 1790564865482051 us; error 39 us; skew 500 ppm
I20260928 03:07:45.482508 20699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:45 dist-test-slave-fjww krb5kdc[20686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564865, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:45.486056 20699 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:07:45.487746 20699 webserver.cc:460] Webserver started at http://127.6.192.62:33859/ using document root <none> and password file <none>
I20260928 03:07:45.488484 20699 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:45.488636 20699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:45.488969 20699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:07:45.492244 20699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "b889c6db2c3549d784c8bfc2906c6e37"
format_stamp: "Formatted at 2026-09-28 03:07:45 on dist-test-slave-fjww"
server_key: "d56b949baf3cfe1b65aa8781b7633d3f"
server_key_iv: "93294f39a4e591357db3fc8c0edecc1a"
server_key_version: "encryptionkey@0"
I20260928 03:07:45.493124 20699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "b889c6db2c3549d784c8bfc2906c6e37"
format_stamp: "Formatted at 2026-09-28 03:07:45 on dist-test-slave-fjww"
server_key: "d56b949baf3cfe1b65aa8781b7633d3f"
server_key_iv: "93294f39a4e591357db3fc8c0edecc1a"
server_key_version: "encryptionkey@0"
I20260928 03:07:45.498905 20699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 03:07:45.503113 20714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:45.504808 20699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:07:45.505096 20699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "b889c6db2c3549d784c8bfc2906c6e37"
format_stamp: "Formatted at 2026-09-28 03:07:45 on dist-test-slave-fjww"
server_key: "d56b949baf3cfe1b65aa8781b7633d3f"
server_key_iv: "93294f39a4e591357db3fc8c0edecc1a"
server_key_version: "encryptionkey@0"
I20260928 03:07:45.505324 20699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:45.528718 20699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:45.533615 20699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:45.533929 20699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:45.554044 20699 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:42845
I20260928 03:07:45.554082 20765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:42845 every 8 connection(s)
I20260928 03:07:45.555732 20699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:07:45.559738 20766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:07:45.564028 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 20699
I20260928 03:07:45.564282 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:07:45.565104 6912 external_mini_cluster.cc:1428] Setting key ff41beb18516d4314f80adab9d491715
I20260928 03:07:45.574474 20766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37: Bootstrap starting.
I20260928 03:07:45.578615 20766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37: Neither blocks nor log segments found. Creating new log.
I20260928 03:07:45.580291 20766 log.cc:824] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37: Log is configured to *not* fsync() on all Append() calls
Sep 28 03:07:45 dist-test-slave-fjww krb5kdc[20686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:07:45.583971 20766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37: No bootstrap required, opened a new log
I20260928 03:07:45.592518 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:45.567125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58220 (local address 127.6.192.62:42845)
0928 03:07:45.567993 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:45.568006 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:45.568049 (+ 43us) server_negotiation.cc:407] Connection header received
0928 03:07:45.569725 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:45.569749 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:45.570204 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:45.570554 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:45.572614 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:45.574051 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:45.576084 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:45.576965 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:45.577114 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:45.582915 (+ 5801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:45.582927 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:45.582952 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:45.583004 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:45.587202 (+ 4198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:45.588222 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:45.588233 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:45.588242 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:45.588469 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:45.588837 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:45.588842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:45.588845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:45.589606 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:45.589929 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:45.590389 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 03:07:45.590888 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":435,"threads_started":1}
I20260928 03:07:45.595381 20766 raft_consensus.cc:348] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b889c6db2c3549d784c8bfc2906c6e37" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 42845 } }
I20260928 03:07:45.595888 20766 raft_consensus.cc:374] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:07:45.596016 20766 raft_consensus.cc:729] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b889c6db2c3549d784c8bfc2906c6e37, State: Initialized, Role: FOLLOWER
I20260928 03:07:45.596681 20766 consensus_queue.cc:260] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b889c6db2c3549d784c8bfc2906c6e37" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 42845 } }
I20260928 03:07:45.596990 20766 raft_consensus.cc:388] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:07:45.597163 20766 raft_consensus.cc:482] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:07:45.597196 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:07:45.597363 20766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:07:45.600772 20766 raft_consensus.cc:504] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b889c6db2c3549d784c8bfc2906c6e37" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 42845 } }
I20260928 03:07:45.601369 20766 leader_election.cc:302] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b889c6db2c3549d784c8bfc2906c6e37; no voters:
I20260928 03:07:45.602077 20766 leader_election.cc:288] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:07:45.602280 20771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:07:45.603190 20771 raft_consensus.cc:686] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37 [term 1 LEADER]: Becoming Leader. State: Replica: b889c6db2c3549d784c8bfc2906c6e37, State: Running, Role: LEADER
I20260928 03:07:45.603863 20771 consensus_queue.cc:237] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b889c6db2c3549d784c8bfc2906c6e37" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 42845 } }
I20260928 03:07:45.604422 20766 sys_catalog.cc:564] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:07:45.607129 20772 sys_catalog.cc:455] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b889c6db2c3549d784c8bfc2906c6e37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b889c6db2c3549d784c8bfc2906c6e37" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 42845 } } }
I20260928 03:07:45.607084 20773 sys_catalog.cc:455] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b889c6db2c3549d784c8bfc2906c6e37. Latest consensus state: current_term: 1 leader_uuid: "b889c6db2c3549d784c8bfc2906c6e37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b889c6db2c3549d784c8bfc2906c6e37" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 42845 } } }
I20260928 03:07:45.607625 20772 sys_catalog.cc:458] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37 [sys.catalog]: This master's current role is: LEADER
I20260928 03:07:45.607630 20773 sys_catalog.cc:458] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37 [sys.catalog]: This master's current role is: LEADER
I20260928 03:07:45.609428 20776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:07:45.615361 20776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:07:45.627151 20776 catalog_manager.cc:1347] Generated new cluster ID: b8717dc189bb4cb1954612db26d5d0db
I20260928 03:07:45.627315 20776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:07:45.651597 20776 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:07:45.653595 20776 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:07:45.662580 20776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b889c6db2c3549d784c8bfc2906c6e37: Generated new TSK 0
I20260928 03:07:45.663919 20776 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:07:45.712128 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 20699
I20260928 03:07:45.713536 20761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:45.819295 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 20699
2026-09-28T03:07:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2136 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:07:45 dist-test-slave-fjww krb5kdc[20794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:07:45 dist-test-slave-fjww krb5kdc[20794](info): set up 2 sockets
Sep 28 03:07:45 dist-test-slave-fjww krb5kdc[20794](info): commencing operation
krb5kdc: starting...
W20260928 03:07:47.044111 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:07:47 dist-test-slave-fjww krb5kdc[20794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:07:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:07:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:07:47.299046 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:43117
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:43117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:07:47.646912 20807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:47.647219 20807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:47.647383 20807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:47.655006 20807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:07:47.655130 20807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:47.655229 20807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:47.655321 20807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:07:47.655367 20807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:07:47.665767 20807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:43117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:43117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:47.667232 20807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:47.668676 20807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:47.681835 20813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:47.681802 20812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:47.682024 20815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:47.682717 20807 server_base.cc:1034] running on GCE node
I20260928 03:07:47.683722 20807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:47.685309 20807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:47.686573 20807 hybrid_clock.cc:648] HybridClock initialized: now 1790564867686525 us; error 41 us; skew 500 ppm
I20260928 03:07:47.686987 20807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:47 dist-test-slave-fjww krb5kdc[20794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564867, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:47.690593 20807 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:07:47.692329 20807 webserver.cc:460] Webserver started at http://127.6.192.62:36881/ using document root <none> and password file <none>
I20260928 03:07:47.693063 20807 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:47.693218 20807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:47.693539 20807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:07:47.696815 20807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "94efea47822443bb9e06dc164a9ec7d1"
format_stamp: "Formatted at 2026-09-28 03:07:47 on dist-test-slave-fjww"
server_key: "b67ef22cf941dfac66e9873184f21ae0"
server_key_iv: "7530e6c64f84457b6cb970ed1c5d8985"
server_key_version: "encryptionkey@0"
I20260928 03:07:47.697737 20807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "94efea47822443bb9e06dc164a9ec7d1"
format_stamp: "Formatted at 2026-09-28 03:07:47 on dist-test-slave-fjww"
server_key: "b67ef22cf941dfac66e9873184f21ae0"
server_key_iv: "7530e6c64f84457b6cb970ed1c5d8985"
server_key_version: "encryptionkey@0"
I20260928 03:07:47.704008 20807 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 03:07:47.708604 20822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:47.710421 20807 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 03:07:47.710675 20807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "94efea47822443bb9e06dc164a9ec7d1"
format_stamp: "Formatted at 2026-09-28 03:07:47 on dist-test-slave-fjww"
server_key: "b67ef22cf941dfac66e9873184f21ae0"
server_key_iv: "7530e6c64f84457b6cb970ed1c5d8985"
server_key_version: "encryptionkey@0"
I20260928 03:07:47.710927 20807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:47.732378 20807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:47.738148 20807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:47.738547 20807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:47.760717 20807 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:43117
I20260928 03:07:47.760740 20873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:43117 every 8 connection(s)
I20260928 03:07:47.762354 20807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:07:47.765134 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 20807
I20260928 03:07:47.765394 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:07:47.766211 6912 external_mini_cluster.cc:1428] Setting key 9c54d806d36bf5864cc3ad1baed830ca
I20260928 03:07:47.767503 20874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 03:07:47 dist-test-slave-fjww krb5kdc[20794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:07:47.783617 20874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1: Bootstrap starting.
I20260928 03:07:47.787427 20874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1: Neither blocks nor log segments found. Creating new log.
I20260928 03:07:47.788642 20874 log.cc:824] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1: Log is configured to *not* fsync() on all Append() calls
I20260928 03:07:47.791764 20874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1: No bootstrap required, opened a new log
I20260928 03:07:47.792802 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:47.768628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42136 (local address 127.6.192.62:43117)
0928 03:07:47.769514 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:47.769526 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:47.769570 (+ 44us) server_negotiation.cc:407] Connection header received
0928 03:07:47.771183 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:47.771209 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:47.771770 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:47.772170 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:47.774096 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:47.775738 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:47.777666 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:47.778523 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:47.778697 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:47.784143 (+ 5446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:47.784154 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:47.784173 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:47.784222 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:47.787918 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:47.788813 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:47.788823 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:47.788834 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:47.789049 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:47.789503 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:47.789508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:47.789513 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:47.790121 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:47.790459 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:47.790958 (+ 499us) server_negotiation.cc:299] Negotiation successful
0928 03:07:47.791666 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":420,"threads_started":1}
I20260928 03:07:47.800397 20874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94efea47822443bb9e06dc164a9ec7d1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43117 } }
I20260928 03:07:47.800943 20874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:07:47.801059 20874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94efea47822443bb9e06dc164a9ec7d1, State: Initialized, Role: FOLLOWER
I20260928 03:07:47.801748 20874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94efea47822443bb9e06dc164a9ec7d1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43117 } }
I20260928 03:07:47.802115 20874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:07:47.802269 20874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:07:47.802451 20874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:07:47.804756 20874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94efea47822443bb9e06dc164a9ec7d1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43117 } }
I20260928 03:07:47.805285 20874 leader_election.cc:302] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94efea47822443bb9e06dc164a9ec7d1; no voters:
I20260928 03:07:47.805932 20874 leader_election.cc:288] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:07:47.806102 20879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:07:47.806855 20879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 1 LEADER]: Becoming Leader. State: Replica: 94efea47822443bb9e06dc164a9ec7d1, State: Running, Role: LEADER
I20260928 03:07:47.807571 20879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94efea47822443bb9e06dc164a9ec7d1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43117 } }
I20260928 03:07:47.808336 20874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:07:47.810933 20880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94efea47822443bb9e06dc164a9ec7d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94efea47822443bb9e06dc164a9ec7d1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43117 } } }
I20260928 03:07:47.811326 20880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [sys.catalog]: This master's current role is: LEADER
I20260928 03:07:47.811033 20881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94efea47822443bb9e06dc164a9ec7d1. Latest consensus state: current_term: 1 leader_uuid: "94efea47822443bb9e06dc164a9ec7d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94efea47822443bb9e06dc164a9ec7d1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43117 } } }
I20260928 03:07:47.811496 20881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [sys.catalog]: This master's current role is: LEADER
I20260928 03:07:47.818176 20886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:07:47.823836 20886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:07:47.834066 20886 catalog_manager.cc:1347] Generated new cluster ID: cc6df311b0f34678bfd179d71b3f972d
I20260928 03:07:47.834218 20886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:07:47.853907 20886 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:07:47.855363 20886 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:07:47.866518 20886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1: Generated new TSK 0
I20260928 03:07:47.867811 20886 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
I20260928 03:07:47.945259 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:43117
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:07:48.306084 20900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:48.306469 20900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:48.306660 20900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:48.315076 20900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:48.315271 20900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:48.315521 20900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:07:48.326295 20900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:43117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:48.327807 20900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:48.329365 20900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:48.344286 20906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:48.344259 20905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:48.344575 20908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:48.345142 20900 server_base.cc:1034] running on GCE node
I20260928 03:07:48.345810 20900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:48.346832 20900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:48.348099 20900 hybrid_clock.cc:648] HybridClock initialized: now 1790564868348061 us; error 38 us; skew 500 ppm
I20260928 03:07:48.348462 20900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:48 dist-test-slave-fjww krb5kdc[20794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564868, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:48.351970 20900 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:07:48.353636 20900 webserver.cc:460] Webserver started at http://127.6.192.1:35729/ using document root <none> and password file <none>
I20260928 03:07:48.354442 20900 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:48.354585 20900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:48.354936 20900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:07:48.358178 20900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "82d60814dc2b49c7a7508d75cf039d18"
format_stamp: "Formatted at 2026-09-28 03:07:48 on dist-test-slave-fjww"
server_key: "28f8b0fdf237d1b3140facb62a189257"
server_key_iv: "ee1e613b4b90ff10d718c5195e8693f4"
server_key_version: "encryptionkey@0"
I20260928 03:07:48.359014 20900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "82d60814dc2b49c7a7508d75cf039d18"
format_stamp: "Formatted at 2026-09-28 03:07:48 on dist-test-slave-fjww"
server_key: "28f8b0fdf237d1b3140facb62a189257"
server_key_iv: "ee1e613b4b90ff10d718c5195e8693f4"
server_key_version: "encryptionkey@0"
I20260928 03:07:48.364454 20900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 03:07:48.368290 20915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:48.369799 20900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:07:48.369998 20900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "82d60814dc2b49c7a7508d75cf039d18"
format_stamp: "Formatted at 2026-09-28 03:07:48 on dist-test-slave-fjww"
server_key: "28f8b0fdf237d1b3140facb62a189257"
server_key_iv: "ee1e613b4b90ff10d718c5195e8693f4"
server_key_version: "encryptionkey@0"
I20260928 03:07:48.370189 20900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:48.408248 20900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:48.413546 20900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:48.413910 20900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:48.415091 20900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:07:48.416879 20900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:07:48.416962 20900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:48.417156 20900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:07:48.417240 20900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:48.455096 20900 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:38953
I20260928 03:07:48.455143 21027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:38953 every 8 connection(s)
I20260928 03:07:48.456818 20900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:07:48.464666 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 20900
I20260928 03:07:48.464916 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:07:48.465814 6912 external_mini_cluster.cc:1428] Setting key 02d29ad7d81dfb993e25869c0032b87d
Sep 28 03:07:48 dist-test-slave-fjww krb5kdc[20794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564868, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:07:48.480548 21030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:48.459687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:60003 (local address 127.6.192.62:43117)
0928 03:07:48.460131 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:48.460135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:48.461275 (+ 1140us) server_negotiation.cc:407] Connection header received
0928 03:07:48.463132 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:48.463138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:48.463243 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:48.463367 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:48.465106 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:48.466440 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:48.468888 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:48.469671 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:48.469874 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:48.473685 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:48.473704 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:48.473712 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:48.473767 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:48.477046 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:48.477950 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:48.477955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:48.477957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:48.478089 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:48.478598 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:48.478602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:48.478604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:48.478884 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:48.479057 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:48.479917 (+ 860us) server_negotiation.cc:299] Negotiation successful
0928 03:07:48.480168 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":233,"threads_started":1}
I20260928 03:07:48.481621 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:48.460318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:60003)
0928 03:07:48.461111 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:48.461155 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:48.462841 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:48.463658 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:48.463667 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:48.464218 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:48.464870 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:48.464898 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:48.466656 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:48.466661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:48.468560 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:48.468574 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:48.469946 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:48.469952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:48.470163 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:48.470919 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:48.470951 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:48.473309 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:48.477234 (+ 3925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:48.477245 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:48.477265 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:48.477761 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:48.478215 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:48.478220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:48.478224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:48.478444 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:48.479029 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:48.479038 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:48.479477 (+ 439us) client_negotiation.cc:769] Sending connection context
0928 03:07:48.479786 (+ 309us) client_negotiation.cc:240] Negotiation successful
0928 03:07:48.480384 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":292,"threads_started":1}
I20260928 03:07:48.483572 21028 heartbeater.cc:347] Connected to a master server at 127.6.192.62:43117
I20260928 03:07:48.483917 21028 heartbeater.cc:464] Registering TS with master...
I20260928 03:07:48.484676 21028 heartbeater.cc:511] Master 127.6.192.62:43117 requested a full tablet report, sending...
I20260928 03:07:48.486975 20839 ts_manager.cc:194] Registered new tserver with Master: 82d60814dc2b49c7a7508d75cf039d18 (127.6.192.1:38953)
I20260928 03:07:48.488562 20839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:60003
WARNING: no policy specified for kudu/127.6.192.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
I20260928 03:07:48.525314 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:43117
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:07:48.885023 21034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:48.885389 21034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:48.885576 21034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:48.893544 21034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:48.893725 21034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:48.893921 21034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:07:48.904330 21034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:43117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:48.905763 21034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:48.907181 21034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:48.923311 21039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:48.923251 21042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:48.923336 21040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:48.929914 21034 server_base.cc:1034] running on GCE node
I20260928 03:07:48.930861 21034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:48.932271 21034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:48.933542 21034 hybrid_clock.cc:648] HybridClock initialized: now 1790564868933474 us; error 62 us; skew 500 ppm
I20260928 03:07:48.933943 21034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:48 dist-test-slave-fjww krb5kdc[20794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564868, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:48.937868 21034 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:07:48.939662 21034 webserver.cc:460] Webserver started at http://127.6.192.2:34567/ using document root <none> and password file <none>
I20260928 03:07:48.940490 21034 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:48.940654 21034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:48.940995 21034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:07:48.944378 21034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "5eaf1fab05534fd9b035ec488cec2486"
format_stamp: "Formatted at 2026-09-28 03:07:48 on dist-test-slave-fjww"
server_key: "2c7e8dca3f0a9b295650abc2b1afc155"
server_key_iv: "ecf7e9e3e94532cbd8eeee6ed75cbb89"
server_key_version: "encryptionkey@0"
I20260928 03:07:48.945272 21034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "5eaf1fab05534fd9b035ec488cec2486"
format_stamp: "Formatted at 2026-09-28 03:07:48 on dist-test-slave-fjww"
server_key: "2c7e8dca3f0a9b295650abc2b1afc155"
server_key_iv: "ecf7e9e3e94532cbd8eeee6ed75cbb89"
server_key_version: "encryptionkey@0"
I20260928 03:07:48.951926 21034 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 03:07:48.955705 21049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:48.957270 21034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:07:48.957522 21034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "5eaf1fab05534fd9b035ec488cec2486"
format_stamp: "Formatted at 2026-09-28 03:07:48 on dist-test-slave-fjww"
server_key: "2c7e8dca3f0a9b295650abc2b1afc155"
server_key_iv: "ecf7e9e3e94532cbd8eeee6ed75cbb89"
server_key_version: "encryptionkey@0"
I20260928 03:07:48.957760 21034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:48.987788 21034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:48.992128 21034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:48.992437 21034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:48.993456 21034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:07:48.994923 21034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:07:48.995029 21034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:48.995175 21034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:07:48.995257 21034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:49.029909 21034 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:35991
I20260928 03:07:49.029960 21161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:35991 every 8 connection(s)
I20260928 03:07:49.031272 21034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:07:49.032829 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 21034
I20260928 03:07:49.033052 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:07:49.033524 6912 external_mini_cluster.cc:1428] Setting key 0654a7e01520b1037c7a81e89b85eb7f
Sep 28 03:07:49 dist-test-slave-fjww krb5kdc[20794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564868, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:07:49.055994 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:49.034543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:45679 (local address 127.6.192.62:43117)
0928 03:07:49.035078 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:49.035085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:49.036186 (+ 1101us) server_negotiation.cc:407] Connection header received
0928 03:07:49.038101 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:49.038111 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:49.038237 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:49.038365 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:49.040034 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.041430 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.043230 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.044044 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.044157 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:49.048192 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:49.048201 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:49.048206 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:49.048256 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:49.052347 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:49.053179 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:49.053186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:49.053190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:49.053362 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:49.053808 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:49.053816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:49.053822 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:49.054239 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:49.054412 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:49.055129 (+ 717us) server_negotiation.cc:299] Negotiation successful
0928 03:07:49.055401 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":270,"threads_started":1}
I20260928 03:07:49.056794 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:49.035132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:45679)
0928 03:07:49.036006 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:49.036053 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:49.037799 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:49.038624 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:49.038633 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:49.039149 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:49.039835 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:49.039856 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.041628 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.041632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:49.043003 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:49.043017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.044339 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.044347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:49.044714 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:49.045575 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:49.045599 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:49.047913 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:49.052537 (+ 4624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:49.052545 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:49.052560 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:49.053021 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:49.053475 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:49.053478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:49.053480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:49.053672 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:49.054367 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:49.054373 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:49.054730 (+ 357us) client_negotiation.cc:769] Sending connection context
0928 03:07:49.054972 (+ 242us) client_negotiation.cc:240] Negotiation successful
0928 03:07:49.055385 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":412,"threads_started":1}
I20260928 03:07:49.058377 21162 heartbeater.cc:347] Connected to a master server at 127.6.192.62:43117
I20260928 03:07:49.058704 21162 heartbeater.cc:464] Registering TS with master...
I20260928 03:07:49.059504 21162 heartbeater.cc:511] Master 127.6.192.62:43117 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.192.3@KRBTEST.COM; defaulting to no policy
I20260928 03:07:49.061055 20839 ts_manager.cc:194] Registered new tserver with Master: 5eaf1fab05534fd9b035ec488cec2486 (127.6.192.2:35991)
I20260928 03:07:49.062117 20839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:45679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
I20260928 03:07:49.091555 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:43117
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:07:49.431413 21168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:49.431828 21168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:49.431931 21168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:49.439687 21168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:49.439815 21168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:49.439929 21168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:07:49.450109 21168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:43117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:49.451501 21168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:49.452903 21168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:49.467507 21173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:49.467473 21174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:49.467530 21176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:49.468567 21168 server_base.cc:1034] running on GCE node
I20260928 03:07:49.469286 21168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:49.470328 21168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:49.471635 21168 hybrid_clock.cc:648] HybridClock initialized: now 1790564869471581 us; error 44 us; skew 500 ppm
I20260928 03:07:49.472033 21168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:49 dist-test-slave-fjww krb5kdc[20794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564869, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:49.475772 21168 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:07:49.477722 21168 webserver.cc:460] Webserver started at http://127.6.192.3:44985/ using document root <none> and password file <none>
I20260928 03:07:49.478546 21168 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:49.478713 21168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:49.479072 21168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:07:49.482522 21168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "d9db0158311340769f8387143a7a36ca"
format_stamp: "Formatted at 2026-09-28 03:07:49 on dist-test-slave-fjww"
server_key: "2cd944cedad41da8a8647e8e394810f6"
server_key_iv: "edd023bee1d9c45d60fd85c273b8dd24"
server_key_version: "encryptionkey@0"
I20260928 03:07:49.483425 21168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "d9db0158311340769f8387143a7a36ca"
format_stamp: "Formatted at 2026-09-28 03:07:49 on dist-test-slave-fjww"
server_key: "2cd944cedad41da8a8647e8e394810f6"
server_key_iv: "edd023bee1d9c45d60fd85c273b8dd24"
server_key_version: "encryptionkey@0"
I20260928 03:07:49.489527 21168 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260928 03:07:49.491737 21028 heartbeater.cc:503] Master 127.6.192.62:43117 was elected leader, sending a full tablet report...
I20260928 03:07:49.494212 21183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:49.495975 21168 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:07:49.496222 21168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "d9db0158311340769f8387143a7a36ca"
format_stamp: "Formatted at 2026-09-28 03:07:49 on dist-test-slave-fjww"
server_key: "2cd944cedad41da8a8647e8e394810f6"
server_key_iv: "edd023bee1d9c45d60fd85c273b8dd24"
server_key_version: "encryptionkey@0"
I20260928 03:07:49.496469 21168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:49.536885 21168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:49.541657 21168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:49.541989 21168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:49.543107 21168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:07:49.544672 21168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:07:49.544795 21168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:49.544956 21168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:07:49.545029 21168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:49.579181 21168 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:32965
I20260928 03:07:49.579231 21295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:32965 every 8 connection(s)
I20260928 03:07:49.580574 21168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:07:49.586493 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 21168
I20260928 03:07:49.586724 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:07:49.587162 6912 external_mini_cluster.cc:1428] Setting key 06f36ee4f0fe3782824e54a413623adc
Sep 28 03:07:49 dist-test-slave-fjww krb5kdc[20794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564869, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:07:49.600884 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:49.583325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:54207 (local address 127.6.192.62:43117)
0928 03:07:49.583830 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:49.583836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:49.584856 (+ 1020us) server_negotiation.cc:407] Connection header received
0928 03:07:49.586732 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:49.586737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:49.586855 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:49.586990 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:49.588645 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.589412 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.590981 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.591384 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.591586 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:49.595018 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:49.595023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:49.595026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:49.595062 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:49.597605 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:49.598386 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:49.598400 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:49.598403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:49.598530 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:49.598957 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:49.598964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:49.598967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:49.599239 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:49.599412 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:49.600149 (+ 737us) server_negotiation.cc:299] Negotiation successful
0928 03:07:49.600387 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":245,"threads_started":1}
I20260928 03:07:49.602020 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:49.583996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:54207)
0928 03:07:49.584685 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:49.584733 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:49.586440 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:49.587242 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:49.587274 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:49.587876 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:49.588442 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:49.588459 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.589571 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.589581 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:49.590817 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:49.590828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.591673 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.591680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:49.591923 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:49.592606 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:49.592634 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:49.594777 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:49.597753 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:49.597763 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:49.597782 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:49.598263 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:49.598643 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:49.598648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:49.598652 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:49.598839 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:49.599367 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:49.599376 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:49.599788 (+ 412us) client_negotiation.cc:769] Sending connection context
0928 03:07:49.600077 (+ 289us) client_negotiation.cc:240] Negotiation successful
0928 03:07:49.600619 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":283,"threads_started":1}
I20260928 03:07:49.603561 21296 heartbeater.cc:347] Connected to a master server at 127.6.192.62:43117
I20260928 03:07:49.603873 21296 heartbeater.cc:464] Registering TS with master...
I20260928 03:07:49.604600 21296 heartbeater.cc:511] Master 127.6.192.62:43117 requested a full tablet report, sending...
I20260928 03:07:49.606355 20838 ts_manager.cc:194] Registered new tserver with Master: d9db0158311340769f8387143a7a36ca (127.6.192.3:32965)
I20260928 03:07:49.607301 20838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:54207
I20260928 03:07:49.616154 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:07:49.635651 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:49.621104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42142 (local address 127.6.192.62:43117)
0928 03:07:49.621311 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:49.621315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:49.621823 (+ 508us) server_negotiation.cc:407] Connection header received
0928 03:07:49.621991 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:49.621995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:49.622065 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:49.622160 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:49.623883 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.624673 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.626302 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.626773 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.626890 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:49.629604 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:49.629609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:49.629613 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:49.629651 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:49.632655 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:49.633373 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:49.633377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:49.633379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:49.633519 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:49.634182 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:49.634186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:49.634188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:49.634529 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:49.634722 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:49.635093 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 03:07:49.635305 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 03:07:49.640556 20838 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:07:49.647859 20838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:07:49.684881 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:49.664163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:38953 (local address 127.0.0.1:38814)
0928 03:07:49.676372 (+ 12209us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:49.676392 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:49.676513 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:49.677464 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:49.677470 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:49.677500 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:07:49.677926 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:49.677942 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.679357 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.679365 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:49.681832 (+ 2467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:49.681846 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.683741 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.683748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:49.683995 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:49.684053 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 03:07:49.684203 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 03:07:49.684422 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12066,"mutex_wait_us":86,"thread_start_us":11914,"threads_started":1}
I20260928 03:07:49.685429 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:49.661153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:32965 (local address 127.0.0.1:42898)
0928 03:07:49.671785 (+ 10632us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:49.671837 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:49.672068 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:49.676417 (+ 4349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:49.676425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:49.676461 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:07:49.676902 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:49.676918 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.679087 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.679095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:49.682331 (+ 3236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:49.682353 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.684531 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.684541 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:49.684726 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:49.684765 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 03:07:49.684859 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 03:07:49.684996 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10355,"mutex_wait_us":261,"thread_start_us":13552,"threads_started":1}
I20260928 03:07:49.686020 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:49.664793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.6.192.1:38953)
0928 03:07:49.675908 (+ 11115us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:49.675916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:49.676653 (+ 737us) server_negotiation.cc:407] Connection header received
0928 03:07:49.676764 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:49.676772 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:49.677029 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:49.677222 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:07:49.678087 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.679163 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.682005 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.683533 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.683677 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:49.683916 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:49.685447 (+ 1531us) server_negotiation.cc:299] Negotiation successful
0928 03:07:49.685661 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10921,"thread_start_us":10746,"threads_started":1}
I20260928 03:07:49.686971 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:49.661396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:35991 (local address 127.0.0.1:48668)
0928 03:07:49.675116 (+ 13720us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:49.675142 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:49.675292 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:49.677157 (+ 1865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:49.677162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:49.677185 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:07:49.677617 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:49.677632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.679524 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.679531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:49.682362 (+ 2831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:49.682371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.686143 (+ 3772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.686150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:49.686326 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:49.686350 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 03:07:49.686471 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 03:07:49.686604 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13548,"mutex_wait_us":146,"thread_start_us":9917,"threads_started":1}
I20260928 03:07:49.686990 21311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:49.661844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48668 (local address 127.6.192.2:35991)
0928 03:07:49.674191 (+ 12347us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:49.674206 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:49.676402 (+ 2196us) server_negotiation.cc:407] Connection header received
0928 03:07:49.676513 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:49.676522 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:49.676917 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:49.677160 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:07:49.677815 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.679332 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.683583 (+ 4251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.685451 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.685805 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:49.686009 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:49.686517 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 03:07:49.686682 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12119,"thread_start_us":12299,"threads_started":1}
I20260928 03:07:49.687009 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:49.661538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42898 (local address 127.6.192.3:32965)
0928 03:07:49.675557 (+ 14019us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:49.675567 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:49.675606 (+ 39us) server_negotiation.cc:407] Connection header received
0928 03:07:49.675718 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:49.675726 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:49.676122 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:49.676997 (+ 875us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:07:49.677307 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.678925 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.682544 (+ 3619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.684058 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.684211 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:49.684447 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:49.685686 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0928 03:07:49.685857 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13770,"thread_start_us":13591,"threads_started":1}
I20260928 03:07:49.690171 21097 tablet_service.cc:1467] Processing CreateTablet for tablet f58d2f8a383e4c57b8c9d3cdaf39f262 (DEFAULT_TABLE table=test-table [id=6cbe92990bed4fbb82bd368570abac74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:07:49.690197 20963 tablet_service.cc:1467] Processing CreateTablet for tablet f58d2f8a383e4c57b8c9d3cdaf39f262 (DEFAULT_TABLE table=test-table [id=6cbe92990bed4fbb82bd368570abac74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:07:49.690578 21231 tablet_service.cc:1467] Processing CreateTablet for tablet f58d2f8a383e4c57b8c9d3cdaf39f262 (DEFAULT_TABLE table=test-table [id=6cbe92990bed4fbb82bd368570abac74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:07:49.691571 21097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f58d2f8a383e4c57b8c9d3cdaf39f262. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:07:49.691668 20963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f58d2f8a383e4c57b8c9d3cdaf39f262. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:07:49.691890 21231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f58d2f8a383e4c57b8c9d3cdaf39f262. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:07:49.704196 21315 tablet_bootstrap.cc:492] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486: Bootstrap starting.
I20260928 03:07:49.706880 21316 tablet_bootstrap.cc:492] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca: Bootstrap starting.
I20260928 03:07:49.707892 21317 tablet_bootstrap.cc:492] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18: Bootstrap starting.
I20260928 03:07:49.708293 21315 tablet_bootstrap.cc:654] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486: Neither blocks nor log segments found. Creating new log.
I20260928 03:07:49.710038 21315 log.cc:824] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486: Log is configured to *not* fsync() on all Append() calls
I20260928 03:07:49.711153 21316 tablet_bootstrap.cc:654] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca: Neither blocks nor log segments found. Creating new log.
I20260928 03:07:49.711735 21317 tablet_bootstrap.cc:654] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18: Neither blocks nor log segments found. Creating new log.
I20260928 03:07:49.712543 21316 log.cc:824] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca: Log is configured to *not* fsync() on all Append() calls
I20260928 03:07:49.712762 21317 log.cc:824] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18: Log is configured to *not* fsync() on all Append() calls
I20260928 03:07:49.713799 21315 tablet_bootstrap.cc:492] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486: No bootstrap required, opened a new log
I20260928 03:07:49.714202 21315 ts_tablet_manager.cc:1397] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 03:07:49.715961 21316 tablet_bootstrap.cc:492] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca: No bootstrap required, opened a new log
I20260928 03:07:49.715965 21317 tablet_bootstrap.cc:492] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18: No bootstrap required, opened a new log
I20260928 03:07:49.716240 21317 ts_tablet_manager.cc:1397] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260928 03:07:49.716272 21316 ts_tablet_manager.cc:1397] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260928 03:07:49.722867 21315 raft_consensus.cc:348] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:49.723326 21315 raft_consensus.cc:374] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:07:49.723424 21315 raft_consensus.cc:729] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eaf1fab05534fd9b035ec488cec2486, State: Initialized, Role: FOLLOWER
I20260928 03:07:49.724165 21315 consensus_queue.cc:260] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:49.725348 21162 heartbeater.cc:503] Master 127.6.192.62:43117 was elected leader, sending a full tablet report...
I20260928 03:07:49.726125 21315 ts_tablet_manager.cc:1428] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486: Time spent starting tablet: real 0.012s user 0.003s sys 0.010s
I20260928 03:07:49.727238 21316 raft_consensus.cc:348] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:49.727754 21316 raft_consensus.cc:374] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:07:49.727854 21316 raft_consensus.cc:729] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9db0158311340769f8387143a7a36ca, State: Initialized, Role: FOLLOWER
I20260928 03:07:49.727691 21317 raft_consensus.cc:348] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:49.728150 21317 raft_consensus.cc:374] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:07:49.728248 21317 raft_consensus.cc:729] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d60814dc2b49c7a7508d75cf039d18, State: Initialized, Role: FOLLOWER
I20260928 03:07:49.728511 21316 consensus_queue.cc:260] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:49.728828 21317 consensus_queue.cc:260] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:49.730374 21296 heartbeater.cc:503] Master 127.6.192.62:43117 was elected leader, sending a full tablet report...
I20260928 03:07:49.731273 21316 ts_tablet_manager.cc:1428] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260928 03:07:49.732069 21317 ts_tablet_manager.cc:1428] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
W20260928 03:07:49.783809 21163 tablet.cc:2369] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:07:49.832383 21297 tablet.cc:2369] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:07:49.880715 21323 raft_consensus.cc:482] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:07:49.881043 21323 raft_consensus.cc:504] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:49.882827 21323 leader_election.cc:288] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9db0158311340769f8387143a7a36ca (127.6.192.3:32965), 5eaf1fab05534fd9b035ec488cec2486 (127.6.192.2:35991)
I20260928 03:07:49.890080 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:49.883182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:57927 (local address 127.6.192.3:32965)
0928 03:07:49.883411 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:49.883417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:49.884203 (+ 786us) server_negotiation.cc:407] Connection header received
0928 03:07:49.884476 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:49.884483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:49.884558 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:49.884625 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:07:49.885313 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.886044 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.888366 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.889188 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.889318 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:49.889421 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:49.889700 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 03:07:49.889843 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 03:07:49.890087 21311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:49.883547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:49413 (local address 127.6.192.2:35991)
0928 03:07:49.883794 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:49.883803 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:49.884342 (+ 539us) server_negotiation.cc:407] Connection header received
0928 03:07:49.884535 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:49.884539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:49.884595 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:49.884704 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:07:49.885346 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.886038 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.888081 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.888977 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.889088 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:49.889163 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:49.889607 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 03:07:49.889745 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 03:07:49.890136 21325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:49.883388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:35991 (local address 127.6.192.1:49413)
0928 03:07:49.884244 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:49.884258 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:49.884387 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:49.884741 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:49.884746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:49.884776 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:07:49.885173 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:49.885181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.886170 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.886178 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:49.887940 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:49.887952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.889221 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.889230 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:49.889432 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:49.889475 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 03:07:49.889549 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 03:07:49.889674 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":53,"thread_start_us":566,"threads_started":1}
I20260928 03:07:49.890419 21324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:49.883190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:32965 (local address 127.6.192.1:57927)
0928 03:07:49.884085 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:49.884101 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:49.884282 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:49.884670 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:49.884677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:49.884704 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:07:49.885168 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:49.885178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.886198 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.886202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:49.888210 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:49.888224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.889397 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.889401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:49.889509 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:49.889562 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 03:07:49.889656 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 03:07:49.889773 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":21,"spinlock_wait_cycles":25472,"thread_start_us":586,"threads_started":1}
I20260928 03:07:49.891309 21117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58d2f8a383e4c57b8c9d3cdaf39f262" candidate_uuid: "82d60814dc2b49c7a7508d75cf039d18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eaf1fab05534fd9b035ec488cec2486" is_pre_election: true
I20260928 03:07:49.891482 21251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58d2f8a383e4c57b8c9d3cdaf39f262" candidate_uuid: "82d60814dc2b49c7a7508d75cf039d18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9db0158311340769f8387143a7a36ca" is_pre_election: true
I20260928 03:07:49.891965 21117 raft_consensus.cc:2454] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82d60814dc2b49c7a7508d75cf039d18 in term 0.
I20260928 03:07:49.891965 21251 raft_consensus.cc:2454] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82d60814dc2b49c7a7508d75cf039d18 in term 0.
I20260928 03:07:49.892688 20919 leader_election.cc:302] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82d60814dc2b49c7a7508d75cf039d18, d9db0158311340769f8387143a7a36ca; no voters:
I20260928 03:07:49.893124 21323 raft_consensus.cc:2781] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:07:49.893272 21323 raft_consensus.cc:482] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:07:49.893410 21323 raft_consensus.cc:3037] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:07:49.895663 21323 raft_consensus.cc:504] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:49.896308 21323 leader_election.cc:288] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 1 election: Requested vote from peers d9db0158311340769f8387143a7a36ca (127.6.192.3:32965), 5eaf1fab05534fd9b035ec488cec2486 (127.6.192.2:35991)
I20260928 03:07:49.896770 21251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58d2f8a383e4c57b8c9d3cdaf39f262" candidate_uuid: "82d60814dc2b49c7a7508d75cf039d18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9db0158311340769f8387143a7a36ca"
I20260928 03:07:49.896879 21117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58d2f8a383e4c57b8c9d3cdaf39f262" candidate_uuid: "82d60814dc2b49c7a7508d75cf039d18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eaf1fab05534fd9b035ec488cec2486"
I20260928 03:07:49.897008 21251 raft_consensus.cc:3037] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:07:49.897135 21117 raft_consensus.cc:3037] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:07:49.899314 21117 raft_consensus.cc:2454] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82d60814dc2b49c7a7508d75cf039d18 in term 1.
I20260928 03:07:49.899322 21251 raft_consensus.cc:2454] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82d60814dc2b49c7a7508d75cf039d18 in term 1.
I20260928 03:07:49.899824 20918 leader_election.cc:302] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5eaf1fab05534fd9b035ec488cec2486, 82d60814dc2b49c7a7508d75cf039d18, d9db0158311340769f8387143a7a36ca; no voters:
I20260928 03:07:49.900208 21323 raft_consensus.cc:2781] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:07:49.900687 21323 raft_consensus.cc:686] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 LEADER]: Becoming Leader. State: Replica: 82d60814dc2b49c7a7508d75cf039d18, State: Running, Role: LEADER
I20260928 03:07:49.901248 21323 consensus_queue.cc:237] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:49.905901 20838 catalog_manager.cc:5495] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82d60814dc2b49c7a7508d75cf039d18 (127.6.192.1). New cstate: current_term: 1 leader_uuid: "82d60814dc2b49c7a7508d75cf039d18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } health_report { overall_health: HEALTHY } } }
I20260928 03:07:49.933470 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:49.927653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38820 (local address 127.6.192.1:38953)
0928 03:07:49.927832 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:49.927837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:49.927855 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:07:49.928055 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:49.928059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:49.928136 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:49.928262 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:07:49.928617 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.929320 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.930786 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.931198 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:49.931293 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:49.931992 (+ 699us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:07:49.932711 (+ 719us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:07:49.932867 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:49.933089 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 03:07:49.933197 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 03:07:49.942315 21251 raft_consensus.cc:1261] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca [term 1 FOLLOWER]: Refusing update from remote peer 82d60814dc2b49c7a7508d75cf039d18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:07:49.942315 21117 raft_consensus.cc:1261] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486 [term 1 FOLLOWER]: Refusing update from remote peer 82d60814dc2b49c7a7508d75cf039d18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:07:49.943418 21326 consensus_queue.cc:1035] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:07:49.943918 21323 consensus_queue.cc:1035] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:07:49.958644 21330 mvcc.cc:205] Tried to move back new op lower bound from 7334153707273216000 to 7334153707119644672. Current Snapshot: MvccSnapshot[applied={T|T < 7334153707273216000 or (T in {7334153707273216000})}]
I20260928 03:07:49.976370 20839 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:07:49.976712 20839 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:07:49.980690 20839 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1: Sending DeleteTablet for 3 replicas of tablet f58d2f8a383e4c57b8c9d3cdaf39f262
I20260928 03:07:49.982064 21231 tablet_service.cc:1514] Processing DeleteTablet for tablet f58d2f8a383e4c57b8c9d3cdaf39f262 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:07:49 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:42898
I20260928 03:07:49.982064 21097 tablet_service.cc:1514] Processing DeleteTablet for tablet f58d2f8a383e4c57b8c9d3cdaf39f262 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:07:49 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:48668
I20260928 03:07:49.982297 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 20900
I20260928 03:07:49.983139 20963 tablet_service.cc:1514] Processing DeleteTablet for tablet f58d2f8a383e4c57b8c9d3cdaf39f262 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:07:49 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:38814
I20260928 03:07:49.999969 21342 tablet_replica.cc:330] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18: stopping tablet replica
I20260928 03:07:50.000495 21342 raft_consensus.cc:2229] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:07:50.001300 21342 raft_consensus.cc:2258] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:07:50.002677 21340 tablet_replica.cc:330] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca: stopping tablet replica
I20260928 03:07:50.003492 21340 raft_consensus.cc:2229] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:07:50.003930 21342 ts_tablet_manager.cc:1905] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:07:50.004319 21340 raft_consensus.cc:2258] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:07:50.005340 21339 tablet_replica.cc:330] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486: stopping tablet replica
I20260928 03:07:50.005975 21339 raft_consensus.cc:2229] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486 [term 1 FOLLOWER]: Raft consensus shutting down.
Sep 28 03:07:49 dist-test-slave-fjww krb5kdc[20794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:07:50.007649 21340 ts_tablet_manager.cc:1905] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:07:50.012916 21342 ts_tablet_manager.cc:1918] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:07:50.013123 21342 log.cc:1196] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/f58d2f8a383e4c57b8c9d3cdaf39f262
I20260928 03:07:50.013723 21342 ts_tablet_manager.cc:1939] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 82d60814dc2b49c7a7508d75cf039d18: Deleting consensus metadata
I20260928 03:07:50.015095 21339 raft_consensus.cc:2258] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:07:50.016335 20823 catalog_manager.cc:4841] TS 82d60814dc2b49c7a7508d75cf039d18 (127.6.192.1:38953): tablet f58d2f8a383e4c57b8c9d3cdaf39f262 (table test-table [id=6cbe92990bed4fbb82bd368570abac74]) successfully deleted
I20260928 03:07:50.018240 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:49.983436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38828 (local address 127.6.192.1:38953)
0928 03:07:49.983637 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:49.983643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:49.994357 (+ 10714us) server_negotiation.cc:407] Connection header received
0928 03:07:49.995324 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:49.995340 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:49.995495 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:49.995578 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:49.997646 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:49.998916 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:50.001198 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:50.001978 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:50.002092 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:50.009157 (+ 7065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:50.009174 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:50.009193 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:50.009245 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:50.014149 (+ 4904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:50.015434 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:50.015471 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:50.015474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:50.015668 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:50.016295 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:50.016300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:50.016303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:50.016853 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:50.017215 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:50.017523 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 03:07:50.017785 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 03:07:50.018260 21339 ts_tablet_manager.cc:1905] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:07:50.019070 21023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:50.019831 21340 ts_tablet_manager.cc:1918] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:07:50.020018 21340 log.cc:1196] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/f58d2f8a383e4c57b8c9d3cdaf39f262
I20260928 03:07:50.020542 21340 ts_tablet_manager.cc:1939] T f58d2f8a383e4c57b8c9d3cdaf39f262 P d9db0158311340769f8387143a7a36ca: Deleting consensus metadata
I20260928 03:07:50.022429 20826 catalog_manager.cc:4841] TS d9db0158311340769f8387143a7a36ca (127.6.192.3:32965): tablet f58d2f8a383e4c57b8c9d3cdaf39f262 (table test-table [id=6cbe92990bed4fbb82bd368570abac74]) successfully deleted
I20260928 03:07:50.026830 21339 ts_tablet_manager.cc:1918] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:07:50.027029 21339 log.cc:1196] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/f58d2f8a383e4c57b8c9d3cdaf39f262
I20260928 03:07:50.027663 21339 ts_tablet_manager.cc:1939] T f58d2f8a383e4c57b8c9d3cdaf39f262 P 5eaf1fab05534fd9b035ec488cec2486: Deleting consensus metadata
I20260928 03:07:50.029727 20825 catalog_manager.cc:4841] TS 5eaf1fab05534fd9b035ec488cec2486 (127.6.192.2:35991): tablet f58d2f8a383e4c57b8c9d3cdaf39f262 (table test-table [id=6cbe92990bed4fbb82bd368570abac74]) successfully deleted
I20260928 03:07:50.121296 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 20900
I20260928 03:07:50.140799 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 21034
Sep 28 03:07:50 dist-test-slave-fjww krb5kdc[20794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.2@KRBTEST.COM
I20260928 03:07:50.157920 21311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:50.141780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48676 (local address 127.6.192.2:35991)
0928 03:07:50.141998 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:50.142004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:50.142105 (+ 101us) server_negotiation.cc:407] Connection header received
0928 03:07:50.142415 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:50.142419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:50.142496 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:50.142606 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:50.143990 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:50.144769 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:50.146069 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:50.146474 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:50.146611 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:50.151597 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:50.151615 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:50.151632 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:50.151676 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:50.154807 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:50.155674 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:50.155682 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:50.155685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:50.155821 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:50.156256 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:50.156260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:50.156262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:50.156670 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:50.156978 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:50.157315 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 03:07:50.157558 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 03:07:50.158711 21157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:50.252590 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 21034
I20260928 03:07:50.270124 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 21168
Sep 28 03:07:50 dist-test-slave-fjww krb5kdc[20794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.3@KRBTEST.COM
I20260928 03:07:50.287531 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:50.270944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42904 (local address 127.6.192.3:32965)
0928 03:07:50.271146 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:50.271154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:50.271178 (+ 24us) server_negotiation.cc:407] Connection header received
0928 03:07:50.271519 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:50.271524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:50.271596 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:50.271719 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:50.273092 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:50.273819 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:50.275202 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:50.275751 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:50.275900 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:50.280745 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:50.280762 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:50.280779 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:50.280843 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:50.284021 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:50.284938 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:50.284947 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:50.284950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:50.285116 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:50.285574 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:50.285580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:50.285583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:50.286036 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:50.286387 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:50.286780 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 03:07:50.287038 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 03:07:50.288237 21291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:50.379745 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 21168
I20260928 03:07:50.396618 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 20807
I20260928 03:07:50.397863 20869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:50.509366 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 20807
I20260928 03:07:50.531042 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:43117
--webserver_interface=127.6.192.62
--webserver_port=36881
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:43117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:07:50.888180 21347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:50.888510 21347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:50.888670 21347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:50.896324 21347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:07:50.896440 21347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:50.896530 21347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:50.896631 21347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:07:50.896694 21347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:07:50.907096 21347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:43117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:43117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=36881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:50.908635 21347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:50.910089 21347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:50.935150 21354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:50.935227 21356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:50.935204 21353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:50.935853 21347 server_base.cc:1034] running on GCE node
I20260928 03:07:50.938107 21347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:50.939917 21347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:50.941198 21347 hybrid_clock.cc:648] HybridClock initialized: now 1790564870941100 us; error 63 us; skew 500 ppm
I20260928 03:07:50.941632 21347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:50 dist-test-slave-fjww krb5kdc[20794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564870, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:50.951417 21347 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:07:50.953450 21347 webserver.cc:460] Webserver started at http://127.6.192.62:36881/ using document root <none> and password file <none>
I20260928 03:07:50.954316 21347 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:50.954499 21347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:50.960999 21347 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 03:07:50.968106 21363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:50.969957 21347 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.007s
I20260928 03:07:50.970189 21347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "94efea47822443bb9e06dc164a9ec7d1"
format_stamp: "Formatted at 2026-09-28 03:07:47 on dist-test-slave-fjww"
server_key: "b67ef22cf941dfac66e9873184f21ae0"
server_key_iv: "7530e6c64f84457b6cb970ed1c5d8985"
server_key_version: "encryptionkey@0"
I20260928 03:07:50.971382 21347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:50.989368 21347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:50.994935 21347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:50.995391 21347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:51.019269 21347 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:43117
I20260928 03:07:51.019304 21414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:43117 every 8 connection(s)
I20260928 03:07:51.021410 21347 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:07:51.026521 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 21347
I20260928 03:07:51.027653 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:38953
--local_ip_for_outbound_sockets=127.6.192.1
--tserver_master_addrs=127.6.192.62:43117
--webserver_port=35729
--webserver_interface=127.6.192.1
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:07:51.031510 21415 sys_catalog.cc:263] Verifying existing consensus state
I20260928 03:07:51.037987 21415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1: Bootstrap starting.
I20260928 03:07:51.060992 21415 log.cc:824] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1: Log is configured to *not* fsync() on all Append() calls
I20260928 03:07:51.074441 21415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 03:07:51.075018 21415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1: Bootstrap complete.
I20260928 03:07:51.084928 21415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94efea47822443bb9e06dc164a9ec7d1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43117 } }
I20260928 03:07:51.085852 21415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94efea47822443bb9e06dc164a9ec7d1, State: Initialized, Role: FOLLOWER
I20260928 03:07:51.086601 21415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94efea47822443bb9e06dc164a9ec7d1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43117 } }
I20260928 03:07:51.086872 21415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:07:51.087009 21415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:07:51.087208 21415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 1 FOLLOWER]: Advancing to term 2
I20260928 03:07:51.091631 21415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94efea47822443bb9e06dc164a9ec7d1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43117 } }
I20260928 03:07:51.092245 21415 leader_election.cc:302] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94efea47822443bb9e06dc164a9ec7d1; no voters:
I20260928 03:07:51.092965 21415 leader_election.cc:288] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 03:07:51.093147 21419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 03:07:51.094636 21419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 2 LEADER]: Becoming Leader. State: Replica: 94efea47822443bb9e06dc164a9ec7d1, State: Running, Role: LEADER
I20260928 03:07:51.095222 21419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94efea47822443bb9e06dc164a9ec7d1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43117 } }
I20260928 03:07:51.095597 21415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:07:51.098052 21421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94efea47822443bb9e06dc164a9ec7d1. Latest consensus state: current_term: 2 leader_uuid: "94efea47822443bb9e06dc164a9ec7d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94efea47822443bb9e06dc164a9ec7d1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43117 } } }
I20260928 03:07:51.098136 21420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "94efea47822443bb9e06dc164a9ec7d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94efea47822443bb9e06dc164a9ec7d1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43117 } } }
I20260928 03:07:51.098526 21421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [sys.catalog]: This master's current role is: LEADER
I20260928 03:07:51.098578 21420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [sys.catalog]: This master's current role is: LEADER
I20260928 03:07:51.109155 21426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:07:51.120242 21426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:07:51.122231 21426 catalog_manager.cc:1259] Loaded cluster ID: cc6df311b0f34678bfd179d71b3f972d
I20260928 03:07:51.122380 21426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:07:51.126946 21426 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 03:07:51.145102 21426 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 03:07:51.146735 21436 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 03:07:51.154706 21426 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:07:51.156586 21426 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1: Loaded TSK: 0
I20260928 03:07:51.158198 21426 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 03:07:51.404114 21417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:51.404413 21417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:51.404561 21417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:51.413018 21417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:51.413141 21417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:51.413318 21417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:07:51.424360 21417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:38953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=35729
--tserver_master_addrs=127.6.192.62:43117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:51.425719 21417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:51.427104 21417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:51.443478 21443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:51.443542 21445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:51.443478 21442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:51.444799 21417 server_base.cc:1034] running on GCE node
I20260928 03:07:51.445626 21417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:51.446805 21417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:51.448094 21417 hybrid_clock.cc:648] HybridClock initialized: now 1790564871448020 us; error 63 us; skew 500 ppm
I20260928 03:07:51.448496 21417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:51 dist-test-slave-fjww krb5kdc[20794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564871, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:51.452260 21417 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:07:51.453982 21417 webserver.cc:460] Webserver started at http://127.6.192.1:35729/ using document root <none> and password file <none>
I20260928 03:07:51.454794 21417 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:51.454942 21417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:51.461692 21417 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.001s
I20260928 03:07:51.465304 21452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:51.466941 21417 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:07:51.467144 21417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "82d60814dc2b49c7a7508d75cf039d18"
format_stamp: "Formatted at 2026-09-28 03:07:48 on dist-test-slave-fjww"
server_key: "28f8b0fdf237d1b3140facb62a189257"
server_key_iv: "ee1e613b4b90ff10d718c5195e8693f4"
server_key_version: "encryptionkey@0"
I20260928 03:07:51.468302 21417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:51.483095 21417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:51.488207 21417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:51.488552 21417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:51.489713 21417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:07:51.491586 21417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:07:51.491710 21417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:51.491859 21417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:07:51.491971 21417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:51.529309 21417 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:38953
I20260928 03:07:51.529320 21565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:38953 every 8 connection(s)
I20260928 03:07:51.530948 21417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:07:51.537720 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 21417
I20260928 03:07:51.538422 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:35991
--local_ip_for_outbound_sockets=127.6.192.2
--tserver_master_addrs=127.6.192.62:43117
--webserver_port=34567
--webserver_interface=127.6.192.2
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:07:51 dist-test-slave-fjww krb5kdc[20794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564871, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:07:51.559463 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:51.534023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:38639 (local address 127.6.192.62:43117)
0928 03:07:51.534677 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:51.534690 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:51.535366 (+ 676us) server_negotiation.cc:407] Connection header received
0928 03:07:51.537551 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:51.537566 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:51.538110 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:51.538557 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:51.540355 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:51.541665 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:51.543348 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:51.544133 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:51.544353 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:51.548328 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:51.548337 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:51.548354 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:51.548399 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:51.552559 (+ 4160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:51.554135 (+ 1576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:51.554148 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:51.554160 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:51.554415 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:51.555076 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:51.555103 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:51.555109 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:51.555944 (+ 835us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:51.556182 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:51.557380 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0928 03:07:51.557992 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":312,"threads_started":1}
I20260928 03:07:51.559477 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:51.534279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:38639)
0928 03:07:51.535173 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:51.535219 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:51.536966 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:51.538756 (+ 1790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:51.538769 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:51.539434 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:51.540081 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:51.540100 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:51.541879 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:51.541883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:51.543169 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:51.543183 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:51.544393 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:51.544401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:51.544694 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:51.545466 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:51.545490 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:51.548012 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:51.552830 (+ 4818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:51.552881 (+ 51us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:51.552903 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:51.553874 (+ 971us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:51.554550 (+ 676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:51.554558 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:51.554565 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:51.554893 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:51.556131 (+ 1238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:51.556145 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:51.556715 (+ 570us) client_negotiation.cc:769] Sending connection context
0928 03:07:51.557074 (+ 359us) client_negotiation.cc:240] Negotiation successful
0928 03:07:51.557768 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":355,"threads_started":1}
I20260928 03:07:51.564108 21566 heartbeater.cc:347] Connected to a master server at 127.6.192.62:43117
I20260928 03:07:51.564478 21566 heartbeater.cc:464] Registering TS with master...
I20260928 03:07:51.565311 21566 heartbeater.cc:511] Master 127.6.192.62:43117 requested a full tablet report, sending...
I20260928 03:07:51.568342 21380 ts_manager.cc:194] Registered new tserver with Master: 82d60814dc2b49c7a7508d75cf039d18 (127.6.192.1:38953)
I20260928 03:07:51.570204 21380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:38639
W20260928 03:07:51.886225 21570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:51.886523 21570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:51.886619 21570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:51.894091 21570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:51.894222 21570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:51.894338 21570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:07:51.904413 21570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:35991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=34567
--tserver_master_addrs=127.6.192.62:43117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:51.905740 21570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:51.907058 21570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:51.921175 21578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:51.921180 21575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:51.921190 21576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:51.921900 21570 server_base.cc:1034] running on GCE node
I20260928 03:07:51.922612 21570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:51.923605 21570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:51.924808 21570 hybrid_clock.cc:648] HybridClock initialized: now 1790564871924747 us; error 59 us; skew 500 ppm
I20260928 03:07:51.925143 21570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:51 dist-test-slave-fjww krb5kdc[20794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564871, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:51.928809 21570 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:07:51.930326 21570 webserver.cc:460] Webserver started at http://127.6.192.2:34567/ using document root <none> and password file <none>
I20260928 03:07:51.931058 21570 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:51.931216 21570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:51.936800 21570 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 03:07:51.940024 21585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:51.941776 21570 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 03:07:51.941967 21570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "5eaf1fab05534fd9b035ec488cec2486"
format_stamp: "Formatted at 2026-09-28 03:07:48 on dist-test-slave-fjww"
server_key: "2c7e8dca3f0a9b295650abc2b1afc155"
server_key_iv: "ecf7e9e3e94532cbd8eeee6ed75cbb89"
server_key_version: "encryptionkey@0"
I20260928 03:07:51.943235 21570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:51.956023 21570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:51.960728 21570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:51.961051 21570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:51.962067 21570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:07:51.963423 21570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:07:51.963605 21570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:51.963743 21570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:07:51.963830 21570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:52.000571 21570 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:35991
I20260928 03:07:52.000669 21697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:35991 every 8 connection(s)
I20260928 03:07:52.002053 21570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:07:52.005230 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 21570
I20260928 03:07:52.006054 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:32965
--local_ip_for_outbound_sockets=127.6.192.3
--tserver_master_addrs=127.6.192.62:43117
--webserver_port=44985
--webserver_interface=127.6.192.3
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:07:52 dist-test-slave-fjww krb5kdc[20794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564871, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:07:52.031178 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.004930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:50533 (local address 127.6.192.62:43117)
0928 03:07:52.005158 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:52.005164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:52.006670 (+ 1506us) server_negotiation.cc:407] Connection header received
0928 03:07:52.010172 (+ 3502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:52.010177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:52.010281 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:52.010427 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:52.013041 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.014208 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.016277 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.017188 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.017591 (+ 403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:52.022875 (+ 5284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:52.022883 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:52.022888 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:52.022942 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:52.026678 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:52.028082 (+ 1404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:52.028089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:52.028093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:52.028254 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:52.028925 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:52.028932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:52.028937 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:52.029295 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:52.029452 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:52.030570 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0928 03:07:52.030833 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 03:07:52.033048 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.005444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:50533)
0928 03:07:52.006458 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:52.006515 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:52.009737 (+ 3222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:52.010789 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:52.010807 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:52.011842 (+ 1035us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:52.012785 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:52.012818 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.014438 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.014448 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:52.016061 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:52.016076 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.017643 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.017648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:52.018075 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:52.019018 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:52.019053 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:52.022529 (+ 3476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:52.026899 (+ 4370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:52.026914 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:52.026940 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:52.027912 (+ 972us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:52.028408 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:52.028415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:52.028423 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:52.028746 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:52.029464 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:52.029478 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:52.030050 (+ 572us) client_negotiation.cc:769] Sending connection context
0928 03:07:52.030453 (+ 403us) client_negotiation.cc:240] Negotiation successful
0928 03:07:52.031207 (+ 754us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":393,"threads_started":1}
I20260928 03:07:52.034816 21698 heartbeater.cc:347] Connected to a master server at 127.6.192.62:43117
I20260928 03:07:52.035209 21698 heartbeater.cc:464] Registering TS with master...
I20260928 03:07:52.036371 21698 heartbeater.cc:511] Master 127.6.192.62:43117 requested a full tablet report, sending...
I20260928 03:07:52.038264 21380 ts_manager.cc:194] Registered new tserver with Master: 5eaf1fab05534fd9b035ec488cec2486 (127.6.192.2:35991)
I20260928 03:07:52.039104 21380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:50533
W20260928 03:07:52.362867 21701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:52.363179 21701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:52.363348 21701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:52.370800 21701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:52.370939 21701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:52.371126 21701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:07:52.381603 21701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:32965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=44985
--tserver_master_addrs=127.6.192.62:43117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:52.382993 21701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:52.384372 21701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:52.398017 21707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:52.397981 21709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:52.398048 21706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:52.398893 21701 server_base.cc:1034] running on GCE node
I20260928 03:07:52.399583 21701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:52.400557 21701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:52.401794 21701 hybrid_clock.cc:648] HybridClock initialized: now 1790564872401746 us; error 43 us; skew 500 ppm
I20260928 03:07:52.402122 21701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:52 dist-test-slave-fjww krb5kdc[20794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564872, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:52.405452 21701 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:07:52.407050 21701 webserver.cc:460] Webserver started at http://127.6.192.3:44985/ using document root <none> and password file <none>
I20260928 03:07:52.407862 21701 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:52.407982 21701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:52.413599 21701 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 03:07:52.416658 21716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:52.418036 21701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 03:07:52.418254 21701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "d9db0158311340769f8387143a7a36ca"
format_stamp: "Formatted at 2026-09-28 03:07:49 on dist-test-slave-fjww"
server_key: "2cd944cedad41da8a8647e8e394810f6"
server_key_iv: "edd023bee1d9c45d60fd85c273b8dd24"
server_key_version: "encryptionkey@0"
I20260928 03:07:52.419256 21701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:52.437492 21701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:52.442202 21701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:52.442502 21701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:52.443593 21701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:07:52.444999 21701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:07:52.445125 21701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:52.445254 21701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:07:52.445350 21701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:52.480715 21701 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:32965
I20260928 03:07:52.480779 21828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:32965 every 8 connection(s)
I20260928 03:07:52.482272 21701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:07:52.486047 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 21701
Sep 28 03:07:52 dist-test-slave-fjww krb5kdc[20794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564872, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:07:52.503876 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.487019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45648 (local address 127.6.192.62:43117)
0928 03:07:52.487566 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:52.487573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:52.487927 (+ 354us) server_negotiation.cc:407] Connection header received
0928 03:07:52.488054 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:52.488062 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:52.488141 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:52.488261 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:52.490006 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.491190 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.493219 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.493869 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.494019 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:52.497300 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:52.497307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:52.497310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:52.497345 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:52.500240 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:52.501157 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:52.501169 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:52.501173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:52.501364 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:52.501814 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:52.501824 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:52.501829 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:52.502272 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:52.502552 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:52.502975 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 03:07:52.503267 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":265,"threads_started":1}
I20260928 03:07:52.507400 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.485234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:50893 (local address 127.6.192.62:43117)
0928 03:07:52.485444 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:52.485449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:52.486816 (+ 1367us) server_negotiation.cc:407] Connection header received
0928 03:07:52.489290 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:52.489297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:52.489381 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:52.489504 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:52.491917 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.493079 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.495291 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.496035 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.496176 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:52.500201 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:52.500206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:52.500209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:52.500240 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:52.503743 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:52.504702 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:52.504724 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:52.504729 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:52.504877 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:52.505420 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:52.505427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:52.505430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:52.505773 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:52.505886 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:52.506976 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0928 03:07:52.507153 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 03:07:52.508653 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.485751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:50893)
0928 03:07:52.486626 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:52.486682 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:52.488985 (+ 2303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:52.489823 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:52.489834 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:52.490685 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:52.491723 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:52.491753 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.493292 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.493298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:52.495115 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:52.495130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.496280 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.496287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:52.496525 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:52.497413 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:52.497438 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:52.499960 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:52.503912 (+ 3952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:52.503920 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:52.503935 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:52.504541 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:52.505005 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:52.505009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:52.505012 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:52.505267 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:52.506044 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:52.506052 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:52.506547 (+ 495us) client_negotiation.cc:769] Sending connection context
0928 03:07:52.506848 (+ 301us) client_negotiation.cc:240] Negotiation successful
0928 03:07:52.507375 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":380,"threads_started":1}
I20260928 03:07:52.510102 21829 heartbeater.cc:347] Connected to a master server at 127.6.192.62:43117
I20260928 03:07:52.510510 21829 heartbeater.cc:464] Registering TS with master...
I20260928 03:07:52.511580 21829 heartbeater.cc:511] Master 127.6.192.62:43117 requested a full tablet report, sending...
I20260928 03:07:52.513271 21380 ts_manager.cc:194] Registered new tserver with Master: d9db0158311340769f8387143a7a36ca (127.6.192.3:32965)
I20260928 03:07:52.514065 21380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:50893
I20260928 03:07:52.520092 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:07:52.527683 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.521087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45650 (local address 127.6.192.62:43117)
0928 03:07:52.521323 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:52.521333 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:52.521682 (+ 349us) server_negotiation.cc:407] Connection header received
0928 03:07:52.521876 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:52.521879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:52.522051 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:52.522178 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:07:52.522625 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.523375 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.525274 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.525716 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.525819 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:52.526406 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:07:52.526937 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:07:52.527048 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:52.527185 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 03:07:52.527260 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 03:07:52.528890 21380 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:07:52.536535 21380 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:07:52.573150 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.549889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58540 (local address 127.6.192.1:38953)
0928 03:07:52.563953 (+ 14064us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:52.563964 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:52.565090 (+ 1126us) server_negotiation.cc:407] Connection header received
0928 03:07:52.565307 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:52.565316 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:52.565585 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:52.565791 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:07:52.566797 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.568395 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.570736 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.571909 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.572061 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:52.572327 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:52.572610 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 03:07:52.572801 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13885,"thread_start_us":13749,"threads_started":1}
I20260928 03:07:52.573411 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.549653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:38953 (local address 127.0.0.1:58540)
0928 03:07:52.564949 (+ 15296us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:52.564965 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:52.565130 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:52.566072 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:52.566076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:52.566107 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:07:52.566536 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:52.566547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.568529 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.568538 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:52.570558 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:52.570573 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.572121 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.572127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:52.572364 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:52.572427 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 03:07:52.572567 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 03:07:52.572859 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15183,"mutex_wait_us":3,"thread_start_us":14391,"threads_started":1}
I20260928 03:07:52.573832 21566 heartbeater.cc:503] Master 127.6.192.62:43117 was elected leader, sending a full tablet report...
I20260928 03:07:52.573863 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.549073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:32965 (local address 127.0.0.1:43302)
0928 03:07:52.561754 (+ 12681us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:52.561799 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:52.562024 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:52.565456 (+ 3432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:52.565460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:52.565507 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:07:52.566054 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:52.566071 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.568356 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.568361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:52.570697 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:52.570718 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.572932 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.572937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:52.573101 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:52.573183 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 03:07:52.573364 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 03:07:52.573495 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12426,"mutex_wait_us":37,"spinlock_wait_cycles":4224,"thread_start_us":15504,"threads_started":1}
I20260928 03:07:52.574590 21841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.549357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43302 (local address 127.6.192.3:32965)
0928 03:07:52.564497 (+ 15140us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:52.564506 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:52.564533 (+ 27us) server_negotiation.cc:407] Connection header received
0928 03:07:52.564643 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:52.564655 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:52.564960 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:52.565164 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:07:52.566443 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.568179 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.570866 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.572455 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.572570 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:52.572787 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:52.574079 (+ 1292us) server_negotiation.cc:299] Negotiation successful
0928 03:07:52.574262 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14960,"thread_start_us":14789,"threads_started":1}
I20260928 03:07:52.575110 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.549869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37452 (local address 127.6.192.2:35991)
0928 03:07:52.560855 (+ 10986us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:52.560869 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:52.564466 (+ 3597us) server_negotiation.cc:407] Connection header received
0928 03:07:52.564791 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:52.564798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:52.564976 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:52.565154 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:07:52.566250 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.567627 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.570570 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.572224 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.572376 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:52.572578 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:52.573507 (+ 929us) server_negotiation.cc:299] Negotiation successful
0928 03:07:52.574484 (+ 977us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10742,"thread_start_us":11024,"threads_started":1}
I20260928 03:07:52.575471 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.549108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:35991 (local address 127.0.0.1:37452)
0928 03:07:52.564331 (+ 15223us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:52.564352 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:52.564599 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:52.565367 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:52.565374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:52.565410 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:07:52.565987 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:52.566009 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.567779 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.567787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:52.570396 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:52.570412 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.572760 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.572768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:52.572963 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:52.573001 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 03:07:52.573087 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 03:07:52.573211 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15081,"mutex_wait_us":87,"spinlock_wait_cycles":7168,"thread_start_us":12234,"threads_started":1}
I20260928 03:07:52.579219 21764 tablet_service.cc:1467] Processing CreateTablet for tablet f7f84861679449cfbe0cdd43a7a6e523 (DEFAULT_TABLE table=test-table [id=b571ad0cb1664c9189ed6cf2dbe662a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:07:52.579211 21500 tablet_service.cc:1467] Processing CreateTablet for tablet f7f84861679449cfbe0cdd43a7a6e523 (DEFAULT_TABLE table=test-table [id=b571ad0cb1664c9189ed6cf2dbe662a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:07:52.579172 21633 tablet_service.cc:1467] Processing CreateTablet for tablet f7f84861679449cfbe0cdd43a7a6e523 (DEFAULT_TABLE table=test-table [id=b571ad0cb1664c9189ed6cf2dbe662a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:07:52.580591 21633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7f84861679449cfbe0cdd43a7a6e523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:07:52.580670 21764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7f84861679449cfbe0cdd43a7a6e523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:07:52.580717 21500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7f84861679449cfbe0cdd43a7a6e523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:07:52.594965 21845 tablet_bootstrap.cc:492] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486: Bootstrap starting.
I20260928 03:07:52.598366 21845 tablet_bootstrap.cc:654] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486: Neither blocks nor log segments found. Creating new log.
I20260928 03:07:52.599515 21845 log.cc:824] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486: Log is configured to *not* fsync() on all Append() calls
I20260928 03:07:52.600260 21846 tablet_bootstrap.cc:492] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18: Bootstrap starting.
I20260928 03:07:52.602941 21847 tablet_bootstrap.cc:492] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca: Bootstrap starting.
I20260928 03:07:52.604426 21845 tablet_bootstrap.cc:492] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486: No bootstrap required, opened a new log
I20260928 03:07:52.604854 21845 ts_tablet_manager.cc:1397] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260928 03:07:52.605664 21846 tablet_bootstrap.cc:654] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18: Neither blocks nor log segments found. Creating new log.
I20260928 03:07:52.606959 21847 tablet_bootstrap.cc:654] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca: Neither blocks nor log segments found. Creating new log.
I20260928 03:07:52.607100 21846 log.cc:824] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18: Log is configured to *not* fsync() on all Append() calls
I20260928 03:07:52.608314 21847 log.cc:824] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca: Log is configured to *not* fsync() on all Append() calls
I20260928 03:07:52.610709 21846 tablet_bootstrap.cc:492] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18: No bootstrap required, opened a new log
I20260928 03:07:52.611047 21846 ts_tablet_manager.cc:1397] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260928 03:07:52.611589 21847 tablet_bootstrap.cc:492] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca: No bootstrap required, opened a new log
I20260928 03:07:52.611861 21847 ts_tablet_manager.cc:1397] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260928 03:07:52.613785 21845 raft_consensus.cc:348] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:52.614411 21845 raft_consensus.cc:374] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:07:52.614529 21845 raft_consensus.cc:729] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eaf1fab05534fd9b035ec488cec2486, State: Initialized, Role: FOLLOWER
I20260928 03:07:52.615171 21845 consensus_queue.cc:260] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:52.616338 21698 heartbeater.cc:503] Master 127.6.192.62:43117 was elected leader, sending a full tablet report...
I20260928 03:07:52.616981 21845 ts_tablet_manager.cc:1428] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260928 03:07:52.622758 21847 raft_consensus.cc:348] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:52.623265 21847 raft_consensus.cc:374] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:07:52.623389 21847 raft_consensus.cc:729] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9db0158311340769f8387143a7a36ca, State: Initialized, Role: FOLLOWER
I20260928 03:07:52.624120 21847 consensus_queue.cc:260] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:52.624431 21846 raft_consensus.cc:348] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:52.624943 21846 raft_consensus.cc:374] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:07:52.625049 21846 raft_consensus.cc:729] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d60814dc2b49c7a7508d75cf039d18, State: Initialized, Role: FOLLOWER
I20260928 03:07:52.625416 21829 heartbeater.cc:503] Master 127.6.192.62:43117 was elected leader, sending a full tablet report...
I20260928 03:07:52.625676 21846 consensus_queue.cc:260] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:52.626098 21847 ts_tablet_manager.cc:1428] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260928 03:07:52.627893 21846 ts_tablet_manager.cc:1428] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
I20260928 03:07:52.720798 21851 raft_consensus.cc:482] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:07:52.721118 21851 raft_consensus.cc:504] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:52.723064 21851 leader_election.cc:288] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9db0158311340769f8387143a7a36ca (127.6.192.3:32965), 82d60814dc2b49c7a7508d75cf039d18 (127.6.192.1:38953)
I20260928 03:07:52.733899 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.723646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:46705 (local address 127.6.192.1:38953)
0928 03:07:52.723860 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:52.723868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:52.724758 (+ 890us) server_negotiation.cc:407] Connection header received
0928 03:07:52.725009 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:52.725014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:52.725089 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:52.725185 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:07:52.726134 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.727241 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.731074 (+ 3833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.732411 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.732566 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:52.732678 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:52.733500 (+ 822us) server_negotiation.cc:299] Negotiation successful
0928 03:07:52.733639 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 03:07:52.733914 21841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.723486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:54985 (local address 127.6.192.3:32965)
0928 03:07:52.723712 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:52.723719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:52.724811 (+ 1092us) server_negotiation.cc:407] Connection header received
0928 03:07:52.725003 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:52.725009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:52.725089 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:52.725185 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:07:52.726134 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.727254 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.731074 (+ 3820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.732429 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.732567 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:52.732678 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:52.733500 (+ 822us) server_negotiation.cc:299] Negotiation successful
0928 03:07:52.733639 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 03:07:52.734156 21854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.723355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:38953 (local address 127.6.192.2:46705)
0928 03:07:52.724623 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:52.724645 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:52.724826 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:52.725218 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:52.725223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:52.725280 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:07:52.725970 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:52.725978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.727427 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.727435 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:52.730837 (+ 3402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:52.730860 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.732931 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.732943 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:52.733232 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:52.733321 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 03:07:52.733434 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 03:07:52.733693 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1122,"mutex_wait_us":378,"spinlock_wait_cycles":33024,"thread_start_us":591,"threads_started":1}
W20260928 03:07:52.734156 21830 tablet.cc:2369] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:07:52.734696 21855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.723525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:32965 (local address 127.6.192.2:54985)
0928 03:07:52.724694 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:52.724711 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:52.724863 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:52.725218 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:52.725223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:52.725269 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:07:52.725962 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:52.725978 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.727429 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.727435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:52.730837 (+ 3402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:52.730860 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.732956 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.732962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:52.733216 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:52.733320 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 03:07:52.733434 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 03:07:52.733697 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"mutex_wait_us":367,"spinlock_wait_cycles":25216,"thread_start_us":732,"threads_started":1}
I20260928 03:07:52.735360 21520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7f84861679449cfbe0cdd43a7a6e523" candidate_uuid: "5eaf1fab05534fd9b035ec488cec2486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d60814dc2b49c7a7508d75cf039d18" is_pre_election: true
I20260928 03:07:52.735618 21784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7f84861679449cfbe0cdd43a7a6e523" candidate_uuid: "5eaf1fab05534fd9b035ec488cec2486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9db0158311340769f8387143a7a36ca" is_pre_election: true
I20260928 03:07:52.735881 21520 raft_consensus.cc:2454] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5eaf1fab05534fd9b035ec488cec2486 in term 0.
I20260928 03:07:52.736148 21784 raft_consensus.cc:2454] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5eaf1fab05534fd9b035ec488cec2486 in term 0.
I20260928 03:07:52.736557 21586 leader_election.cc:302] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5eaf1fab05534fd9b035ec488cec2486, 82d60814dc2b49c7a7508d75cf039d18; no voters:
I20260928 03:07:52.737001 21851 raft_consensus.cc:2781] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:07:52.737149 21851 raft_consensus.cc:482] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:07:52.737262 21851 raft_consensus.cc:3037] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:07:52.739506 21851 raft_consensus.cc:504] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:52.740180 21851 leader_election.cc:288] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [CANDIDATE]: Term 1 election: Requested vote from peers d9db0158311340769f8387143a7a36ca (127.6.192.3:32965), 82d60814dc2b49c7a7508d75cf039d18 (127.6.192.1:38953)
I20260928 03:07:52.740600 21784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7f84861679449cfbe0cdd43a7a6e523" candidate_uuid: "5eaf1fab05534fd9b035ec488cec2486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9db0158311340769f8387143a7a36ca"
I20260928 03:07:52.740723 21520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7f84861679449cfbe0cdd43a7a6e523" candidate_uuid: "5eaf1fab05534fd9b035ec488cec2486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d60814dc2b49c7a7508d75cf039d18"
I20260928 03:07:52.740821 21784 raft_consensus.cc:3037] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:07:52.740973 21520 raft_consensus.cc:3037] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:07:52.743183 21784 raft_consensus.cc:2454] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eaf1fab05534fd9b035ec488cec2486 in term 1.
I20260928 03:07:52.743304 21520 raft_consensus.cc:2454] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eaf1fab05534fd9b035ec488cec2486 in term 1.
I20260928 03:07:52.743702 21589 leader_election.cc:302] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5eaf1fab05534fd9b035ec488cec2486, d9db0158311340769f8387143a7a36ca; no voters:
I20260928 03:07:52.744047 21851 raft_consensus.cc:2781] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:07:52.744510 21851 raft_consensus.cc:686] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [term 1 LEADER]: Becoming Leader. State: Replica: 5eaf1fab05534fd9b035ec488cec2486, State: Running, Role: LEADER
I20260928 03:07:52.745002 21851 consensus_queue.cc:237] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:52.750118 21379 catalog_manager.cc:5495] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5eaf1fab05534fd9b035ec488cec2486 (127.6.192.2). New cstate: current_term: 1 leader_uuid: "5eaf1fab05534fd9b035ec488cec2486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } health_report { overall_health: UNKNOWN } } }
W20260928 03:07:52.753989 21699 tablet.cc:2369] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:07:52.769625 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.763621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37460 (local address 127.6.192.2:35991)
0928 03:07:52.763869 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:52.763873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:52.763891 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:07:52.764011 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:52.764014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:52.764090 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:52.764222 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:07:52.764568 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.765395 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.767151 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.767677 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.767883 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:52.768370 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:07:52.769010 (+ 640us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:07:52.769142 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:52.769225 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 03:07:52.769299 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 03:07:52.778422 21520 raft_consensus.cc:1261] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Refusing update from remote peer 5eaf1fab05534fd9b035ec488cec2486: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:07:52.778481 21784 raft_consensus.cc:1261] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca [term 1 FOLLOWER]: Refusing update from remote peer 5eaf1fab05534fd9b035ec488cec2486: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:07:52.779491 21856 consensus_queue.cc:1035] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:07:52.779886 21851 consensus_queue.cc:1035] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 03:07:52.783752 21567 tablet.cc:2369] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:07:52.811686 21380 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:07:52.812017 21380 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:07:52.816094 21380 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1: Sending DeleteTablet for 3 replicas of tablet f7f84861679449cfbe0cdd43a7a6e523
I20260928 03:07:52.817629 21764 tablet_service.cc:1514] Processing DeleteTablet for tablet f7f84861679449cfbe0cdd43a7a6e523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:07:52 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:43302
I20260928 03:07:52.817639 21500 tablet_service.cc:1514] Processing DeleteTablet for tablet f7f84861679449cfbe0cdd43a7a6e523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:07:52 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:58540
I20260928 03:07:52.818149 21633 tablet_service.cc:1514] Processing DeleteTablet for tablet f7f84861679449cfbe0cdd43a7a6e523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:07:52 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:37452
I20260928 03:07:52.819608 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 21417
I20260928 03:07:52.829921 21870 tablet_replica.cc:330] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486: stopping tablet replica
I20260928 03:07:52.830734 21870 raft_consensus.cc:2229] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:07:52.831149 21869 tablet_replica.cc:330] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca: stopping tablet replica
I20260928 03:07:52.831760 21870 raft_consensus.cc:2258] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:07:52.832649 21871 tablet_replica.cc:330] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18: stopping tablet replica
I20260928 03:07:52.833117 21871 raft_consensus.cc:2229] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:07:52.833067 21869 raft_consensus.cc:2229] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:07:52.833653 21871 raft_consensus.cc:2258] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:07:52.833673 21869 raft_consensus.cc:2258] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:07:52.835999 21870 ts_tablet_manager.cc:1905] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:07:52.836601 21869 ts_tablet_manager.cc:1905] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:07:52.837882 21871 ts_tablet_manager.cc:1905] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:07:52.845042 21869 ts_tablet_manager.cc:1918] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:07:52.845302 21869 log.cc:1196] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/f7f84861679449cfbe0cdd43a7a6e523
I20260928 03:07:52.845455 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.820632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58548 (local address 127.6.192.1:38953)
0928 03:07:52.821416 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:52.821425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:52.821450 (+ 25us) server_negotiation.cc:407] Connection header received
0928 03:07:52.821528 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:52.821536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:52.821686 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:52.821821 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:52.825157 (+ 3336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.827034 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.830172 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.831115 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.831262 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:52.835679 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:52.835699 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:52.835718 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:52.835847 (+ 129us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:52.840352 (+ 4505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:52.841593 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:52.841603 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:52.841606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:52.841807 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:52.842340 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:52.842345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:52.842349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:52.842945 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:52.843268 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:52.844704 (+ 1436us) server_negotiation.cc:299] Negotiation successful
0928 03:07:52.845011 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587}
I20260928 03:07:52.845959 21869 ts_tablet_manager.cc:1939] T f7f84861679449cfbe0cdd43a7a6e523 P d9db0158311340769f8387143a7a36ca: Deleting consensus metadata
I20260928 03:07:52.846625 21561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:52.847152 21870 ts_tablet_manager.cc:1918] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:07:52.847466 21870 log.cc:1196] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/f7f84861679449cfbe0cdd43a7a6e523
I20260928 03:07:52.848155 21870 ts_tablet_manager.cc:1939] T f7f84861679449cfbe0cdd43a7a6e523 P 5eaf1fab05534fd9b035ec488cec2486: Deleting consensus metadata
I20260928 03:07:52.848508 21367 catalog_manager.cc:4841] TS d9db0158311340769f8387143a7a36ca (127.6.192.3:32965): tablet f7f84861679449cfbe0cdd43a7a6e523 (table test-table [id=b571ad0cb1664c9189ed6cf2dbe662a0]) successfully deleted
I20260928 03:07:52.850857 21366 catalog_manager.cc:4841] TS 5eaf1fab05534fd9b035ec488cec2486 (127.6.192.2:35991): tablet f7f84861679449cfbe0cdd43a7a6e523 (table test-table [id=b571ad0cb1664c9189ed6cf2dbe662a0]) successfully deleted
I20260928 03:07:52.967366 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 21417
W20260928 03:07:52.985369 21364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.192.1:38953 (error 108)
I20260928 03:07:52.985523 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 21570
I20260928 03:07:52.989296 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.988847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:38953 (local address 127.0.0.1:58560)
0928 03:07:52.989033 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:52.989150 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:38953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 03:07:52.989634 21364 catalog_manager.cc:4584] TS 82d60814dc2b49c7a7508d75cf039d18 (127.6.192.1:38953): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7f84861679449cfbe0cdd43a7a6e523: Network error: Client connection negotiation failed: client connection to 127.6.192.1:38953: connect: Connection refused (error 111)
I20260928 03:07:53.003311 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:52.986360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37470 (local address 127.6.192.2:35991)
0928 03:07:52.986551 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:52.986558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:52.986666 (+ 108us) server_negotiation.cc:407] Connection header received
0928 03:07:52.986875 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:52.986881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:52.986990 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:52.987096 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:52.988674 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.989962 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.991655 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:52.992288 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:52.992436 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:52.995271 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:52.995291 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:52.995311 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:52.995362 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:52.999722 (+ 4360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:53.000747 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:53.000757 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:53.000761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:53.000966 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:53.001408 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:53.001413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:53.001417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:53.001902 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:53.002278 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:53.002638 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 03:07:53.002884 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 03:07:53.004163 21693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:53.043578 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:53.043014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:38953 (local address 127.0.0.1:58572)
0928 03:07:53.043240 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:53.043344 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:38953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 03:07:53.044778 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:53.044362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:38953 (local address 127.0.0.1:58576)
0928 03:07:53.044539 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:53.044646 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:38953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 03:07:53.105727 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:53.105195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:38953 (local address 127.0.0.1:58578)
0928 03:07:53.105459 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:53.105552 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:38953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 03:07:53.107070 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:53.106591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:38953 (local address 127.0.0.1:58594)
0928 03:07:53.106860 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:53.106941 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:38953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 03:07:53.133448 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 21570
I20260928 03:07:53.152100 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 21701
I20260928 03:07:53.167074 21841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:53.152837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.6.192.3:32965)
0928 03:07:53.153048 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:53.153054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:53.153075 (+ 21us) server_negotiation.cc:407] Connection header received
0928 03:07:53.153258 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:53.153264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:53.153351 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:53.153462 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:53.154832 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:53.155648 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:53.156963 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:53.157364 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:53.157498 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:53.160703 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:53.160720 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:53.160736 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:53.160779 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:53.164192 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:53.165051 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:53.165059 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:53.165061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:53.165267 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:53.165627 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:53.165630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:53.165632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:53.166031 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:53.166300 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:53.166581 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 03:07:53.166796 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 03:07:53.167842 21824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:53.187866 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:53.187218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:38953 (local address 127.0.0.1:58596)
0928 03:07:53.187499 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:53.187613 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:38953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 03:07:53.189014 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:53.188569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:38953 (local address 127.0.0.1:58606)
0928 03:07:53.188791 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:53.188886 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:38953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260928 03:07:53.259548 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 21701
I20260928 03:07:53.276132 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 21347
I20260928 03:07:53.277457 21410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:07:53.395012 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 21347
I20260928 03:07:53.395062 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:53.394205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:38953 (local address 127.0.0.1:58618)
0928 03:07:53.394522 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:53.394881 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:38953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260928 03:07:53.416874 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:43117
--webserver_interface=127.6.192.62
--webserver_port=36881
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:43117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:07:53.761045 21877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:53.761358 21877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:53.761454 21877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:53.768908 21877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:07:53.769034 21877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:53.769083 21877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:53.769133 21877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:07:53.769169 21877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:07:53.779289 21877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:43117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:43117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=36881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:53.780622 21877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:53.782086 21877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:53.795512 21883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:53.795503 21885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:53.795854 21882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:53.801273 21877 server_base.cc:1034] running on GCE node
I20260928 03:07:53.802330 21877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:53.803967 21877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:53.805229 21877 hybrid_clock.cc:648] HybridClock initialized: now 1790564873805165 us; error 28 us; skew 500 ppm
I20260928 03:07:53.805697 21877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:53 dist-test-slave-fjww krb5kdc[20794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564873, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:53.809103 21877 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:07:53.810745 21877 webserver.cc:460] Webserver started at http://127.6.192.62:36881/ using document root <none> and password file <none>
I20260928 03:07:53.811616 21877 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:53.811779 21877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:53.817720 21877 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 03:07:53.821244 21892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:53.823024 21877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 03:07:53.823246 21877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "94efea47822443bb9e06dc164a9ec7d1"
format_stamp: "Formatted at 2026-09-28 03:07:47 on dist-test-slave-fjww"
server_key: "b67ef22cf941dfac66e9873184f21ae0"
server_key_iv: "7530e6c64f84457b6cb970ed1c5d8985"
server_key_version: "encryptionkey@0"
I20260928 03:07:53.824465 21877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:53.850641 21877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:53.855258 21877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:53.855597 21877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:53.876045 21877 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:43117
I20260928 03:07:53.876089 21943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:43117 every 8 connection(s)
I20260928 03:07:53.877702 21877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:07:53.883670 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 21877
I20260928 03:07:53.884796 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:38953
--local_ip_for_outbound_sockets=127.6.192.1
--tserver_master_addrs=127.6.192.62:43117
--webserver_port=35729
--webserver_interface=127.6.192.1
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:07:53.886191 21944 sys_catalog.cc:263] Verifying existing consensus state
I20260928 03:07:53.892280 21944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1: Bootstrap starting.
I20260928 03:07:53.915930 21944 log.cc:824] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1: Log is configured to *not* fsync() on all Append() calls
I20260928 03:07:53.935035 21944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 03:07:53.935739 21944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1: Bootstrap complete.
I20260928 03:07:53.946611 21944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94efea47822443bb9e06dc164a9ec7d1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43117 } }
I20260928 03:07:53.948436 21944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94efea47822443bb9e06dc164a9ec7d1, State: Initialized, Role: FOLLOWER
I20260928 03:07:53.949237 21944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94efea47822443bb9e06dc164a9ec7d1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43117 } }
I20260928 03:07:53.949568 21944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:07:53.949719 21944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:07:53.949944 21944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 2 FOLLOWER]: Advancing to term 3
I20260928 03:07:53.954471 21944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94efea47822443bb9e06dc164a9ec7d1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43117 } }
I20260928 03:07:53.955057 21944 leader_election.cc:302] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94efea47822443bb9e06dc164a9ec7d1; no voters:
I20260928 03:07:53.955909 21944 leader_election.cc:288] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 03:07:53.956073 21948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 03:07:53.957329 21948 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [term 3 LEADER]: Becoming Leader. State: Replica: 94efea47822443bb9e06dc164a9ec7d1, State: Running, Role: LEADER
I20260928 03:07:53.958050 21948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94efea47822443bb9e06dc164a9ec7d1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43117 } }
I20260928 03:07:53.958350 21944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:07:53.960860 21950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94efea47822443bb9e06dc164a9ec7d1. Latest consensus state: current_term: 3 leader_uuid: "94efea47822443bb9e06dc164a9ec7d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94efea47822443bb9e06dc164a9ec7d1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43117 } } }
I20260928 03:07:53.961365 21950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [sys.catalog]: This master's current role is: LEADER
I20260928 03:07:53.962050 21949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "94efea47822443bb9e06dc164a9ec7d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94efea47822443bb9e06dc164a9ec7d1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 43117 } } }
I20260928 03:07:53.962442 21949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94efea47822443bb9e06dc164a9ec7d1 [sys.catalog]: This master's current role is: LEADER
I20260928 03:07:53.966562 21954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:07:53.979326 21954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:07:53.981832 21954 catalog_manager.cc:1259] Loaded cluster ID: cc6df311b0f34678bfd179d71b3f972d
I20260928 03:07:53.982118 21954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 03:07:53.985307 21954 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6fa16d09ad google::LogMessage::Fail() at ??:0
@ 0x7f6fa16d388a google::LogMessage::SendToLog() at ??:0
@ 0x7f6fa16d04de google::LogMessage::Flush() at ??:0
@ 0x7f6fa16d1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6fb7103512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f6fb7100976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6fb7188610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f6fb7188439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
I20260928 03:07:54.032222 21969 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f6fa431c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6fa431f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f6fa431f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6fa42faeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6fa75d46db start_thread at ??:0
@ 0x7f6fa056a61f clone at ??:0
W20260928 03:07:54.362676 21946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:54.363059 21946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:54.363253 21946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:54.371019 21946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:54.371235 21946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:54.371503 21946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:07:54.382143 21946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:38953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=35729
--tserver_master_addrs=127.6.192.62:43117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:54.383826 21946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:54.385351 21946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:54.400054 21989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:54.400038 21986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:54.400161 21987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:54.400905 21946 server_base.cc:1034] running on GCE node
I20260928 03:07:54.401578 21946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:54.402619 21946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:54.403856 21946 hybrid_clock.cc:648] HybridClock initialized: now 1790564874403785 us; error 67 us; skew 500 ppm
I20260928 03:07:54.404254 21946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:54 dist-test-slave-fjww krb5kdc[20794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564874, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:54.407673 21946 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:07:54.409337 21946 webserver.cc:460] Webserver started at http://127.6.192.1:35729/ using document root <none> and password file <none>
I20260928 03:07:54.410089 21946 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:54.410199 21946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:54.416256 21946 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 03:07:54.419842 21996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:54.421389 21946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 03:07:54.421587 21946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "82d60814dc2b49c7a7508d75cf039d18"
format_stamp: "Formatted at 2026-09-28 03:07:48 on dist-test-slave-fjww"
server_key: "28f8b0fdf237d1b3140facb62a189257"
server_key_iv: "ee1e613b4b90ff10d718c5195e8693f4"
server_key_version: "encryptionkey@0"
I20260928 03:07:54.422645 21946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:54.447207 21946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:54.452363 21946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:54.452677 21946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:54.453797 21946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:07:54.455947 22003 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260928 03:07:54.462009 21946 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 03:07:54.462139 21946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260928 03:07:54.462297 21946 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 03:07:54.465791 21946 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 03:07:54.465912 21946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.002s
I20260928 03:07:54.466394 22003 tablet_bootstrap.cc:492] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18: Bootstrap starting.
I20260928 03:07:54.476603 22003 log.cc:824] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18: Log is configured to *not* fsync() on all Append() calls
I20260928 03:07:54.497702 22003 tablet_bootstrap.cc:492] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 03:07:54.498557 22003 tablet_bootstrap.cc:492] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18: Bootstrap complete.
I20260928 03:07:54.499244 22003 ts_tablet_manager.cc:1397] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.009s
I20260928 03:07:54.508644 21946 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:38953
I20260928 03:07:54.508672 22110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:38953 every 8 connection(s)
I20260928 03:07:54.510455 21946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:07:54.512606 22003 raft_consensus.cc:348] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:54.513688 22003 raft_consensus.cc:729] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d60814dc2b49c7a7508d75cf039d18, State: Initialized, Role: FOLLOWER
I20260928 03:07:54.514416 22003 consensus_queue.cc:260] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:54.516150 22003 ts_tablet_manager.cc:1428] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18: Time spent starting tablet: real 0.017s user 0.020s sys 0.000s
I20260928 03:07:54.516317 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:54.513821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:56703)
0928 03:07:54.514830 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:54.515149 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":448,"threads_started":1}
I20260928 03:07:54.516721 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 21946
W20260928 03:07:54.517148 22111 heartbeater.cc:644] Failed to heartbeat to 127.6.192.62:43117 (0 consecutive failures): Network error: Failed to ping master at 127.6.192.62:43117: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
I20260928 03:07:54.517588 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:35991
--local_ip_for_outbound_sockets=127.6.192.2
--tserver_master_addrs=127.6.192.62:43117
--webserver_port=34567
--webserver_interface=127.6.192.2
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:07:54.518278 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:54.517832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:39105)
0928 03:07:54.518012 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:54.518112 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 03:07:54.519402 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:54.518933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:45145)
0928 03:07:54.519113 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:54.519218 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 03:07:54.519692 22111 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 03:07:54.884335 22115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:54.884639 22115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:54.884737 22115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:54.892297 22115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:54.892421 22115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:54.892541 22115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:07:54.902540 22115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:35991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=34567
--tserver_master_addrs=127.6.192.62:43117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:54.903959 22115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:54.905272 22115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:54.919858 22120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:54.919977 22121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:54.919947 22123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:54.921556 22115 server_base.cc:1034] running on GCE node
I20260928 03:07:54.922276 22115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:54.923369 22115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:54.924681 22115 hybrid_clock.cc:648] HybridClock initialized: now 1790564874924589 us; error 82 us; skew 500 ppm
I20260928 03:07:54.925078 22115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:54 dist-test-slave-fjww krb5kdc[20794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564874, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:54.928743 22115 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:07:54.930442 22115 webserver.cc:460] Webserver started at http://127.6.192.2:34567/ using document root <none> and password file <none>
I20260928 03:07:54.931190 22115 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:54.931358 22115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:54.937184 22115 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:07:54.940076 22130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:54.941596 22115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:07:54.941794 22115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "5eaf1fab05534fd9b035ec488cec2486"
format_stamp: "Formatted at 2026-09-28 03:07:48 on dist-test-slave-fjww"
server_key: "2c7e8dca3f0a9b295650abc2b1afc155"
server_key_iv: "ecf7e9e3e94532cbd8eeee6ed75cbb89"
server_key_version: "encryptionkey@0"
I20260928 03:07:54.942901 22115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:54.968004 22115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:54.972659 22115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:54.972966 22115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:54.974027 22115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:07:54.975489 22115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:07:54.975620 22115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:54.975744 22115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:07:54.975832 22115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:55.012130 22115 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:35991
I20260928 03:07:55.012183 22242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:35991 every 8 connection(s)
I20260928 03:07:55.013652 22115 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:07:55.014657 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 22115
I20260928 03:07:55.015579 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:32965
--local_ip_for_outbound_sockets=127.6.192.3
--tserver_master_addrs=127.6.192.62:43117
--webserver_port=44985
--webserver_interface=127.6.192.3
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:07:55.019950 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:55.017370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:38197)
0928 03:07:55.018300 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:55.018610 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":358,"threads_started":1}
W20260928 03:07:55.021020 22243 heartbeater.cc:644] Failed to heartbeat to 127.6.192.62:43117 (0 consecutive failures): Network error: Failed to ping master at 127.6.192.62:43117: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
I20260928 03:07:55.025336 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:55.021704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:47871)
0928 03:07:55.025045 (+ 3341us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:55.025153 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3175,"thread_start_us":3310,"threads_started":1}
I20260928 03:07:55.026521 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:55.026025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:49903)
0928 03:07:55.026261 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:55.026383 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 03:07:55.026803 22243 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 03:07:55.360812 22247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:07:55.361166 22247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:07:55.361340 22247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:07:55.368883 22247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:07:55.369058 22247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:07:55.369253 22247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:07:55.379667 22247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:32965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=44985
--tserver_master_addrs=127.6.192.62:43117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:07:55.381073 22247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:07:55.382552 22247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:07:55.397526 22255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:55.397547 22253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:07:55.397583 22252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:07:55.398526 22247 server_base.cc:1034] running on GCE node
I20260928 03:07:55.399183 22247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:07:55.400269 22247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:07:55.401522 22247 hybrid_clock.cc:648] HybridClock initialized: now 1790564875401451 us; error 67 us; skew 500 ppm
I20260928 03:07:55.401894 22247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:07:55 dist-test-slave-fjww krb5kdc[20794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564875, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:07:55.405558 22247 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:07:55.407162 22247 webserver.cc:460] Webserver started at http://127.6.192.3:44985/ using document root <none> and password file <none>
I20260928 03:07:55.408007 22247 fs_manager.cc:359] Metadata directory not provided
I20260928 03:07:55.408143 22247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:07:55.413859 22247 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 03:07:55.417068 22262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:55.418646 22247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:07:55.418792 22247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "d9db0158311340769f8387143a7a36ca"
format_stamp: "Formatted at 2026-09-28 03:07:49 on dist-test-slave-fjww"
server_key: "2cd944cedad41da8a8647e8e394810f6"
server_key_iv: "edd023bee1d9c45d60fd85c273b8dd24"
server_key_version: "encryptionkey@0"
I20260928 03:07:55.419983 22247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:07:55.436815 22247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:07:55.441320 22247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:07:55.441622 22247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:07:55.442706 22247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:07:55.444267 22247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:07:55.444370 22247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:55.444489 22247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:07:55.444557 22247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:07:55.483713 22247 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:32965
I20260928 03:07:55.483805 22374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:32965 every 8 connection(s)
I20260928 03:07:55.485347 22247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:07:55.491204 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:55.489006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:45577)
0928 03:07:55.489873 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:55.490096 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":351,"threads_started":1}
W20260928 03:07:55.492130 22375 heartbeater.cc:644] Failed to heartbeat to 127.6.192.62:43117 (0 consecutive failures): Network error: Failed to ping master at 127.6.192.62:43117: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
I20260928 03:07:55.493335 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:55.492957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:53819)
0928 03:07:55.493127 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:55.493210 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 03:07:55.494490 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:55.494104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:60879)
0928 03:07:55.494262 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:55.494353 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 03:07:55.495000 22375 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 03:07:55.495114 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 22247
I20260928 03:07:55.521199 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:55.520386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:50683)
0928 03:07:55.520883 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:55.520984 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":223,"threads_started":1}
I20260928 03:07:55.828181 22380 raft_consensus.cc:482] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:07:55.828501 22380 raft_consensus.cc:504] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:55.830200 22380 leader_election.cc:288] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9db0158311340769f8387143a7a36ca (127.6.192.3:32965), 5eaf1fab05534fd9b035ec488cec2486 (127.6.192.2:35991)
Sep 28 03:07:55 dist-test-slave-fjww krb5kdc[20794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564874, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.2@KRBTEST.COM
Sep 28 03:07:55 dist-test-slave-fjww krb5kdc[20794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564874, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.3@KRBTEST.COM
I20260928 03:07:55.858534 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:55.831069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:51271 (local address 127.6.192.2:35991)
0928 03:07:55.833701 (+ 2632us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:55.833725 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:55.834522 (+ 797us) server_negotiation.cc:407] Connection header received
0928 03:07:55.836138 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:55.836146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:55.836526 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:55.836901 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:55.838850 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:55.840231 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:55.842159 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:55.842815 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:55.843057 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:55.847776 (+ 4719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:55.847790 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:55.847814 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:55.847934 (+ 120us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:55.852602 (+ 4668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:55.854253 (+ 1651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:55.854264 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:55.854274 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:55.854566 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:55.855203 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:55.855207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:55.855211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:55.856048 (+ 837us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:55.856239 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:55.857597 (+ 1358us) server_negotiation.cc:299] Negotiation successful
0928 03:07:55.858066 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2388,"thread_start_us":2191,"threads_started":1}
I20260928 03:07:55.858534 22383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:55.831226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:50415 (local address 127.6.192.3:32965)
0928 03:07:55.834174 (+ 2948us) server_negotiation.cc:206] Beginning negotiation
0928 03:07:55.834186 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:07:55.834227 (+ 41us) server_negotiation.cc:407] Connection header received
0928 03:07:55.835853 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:07:55.835862 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:07:55.836289 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:07:55.836589 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:07:55.838919 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:55.840324 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:55.842158 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:07:55.842817 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:07:55.843026 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:55.847945 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:07:55.847959 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:07:55.847975 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:07:55.848026 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:07:55.852602 (+ 4576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:55.854253 (+ 1651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:55.854264 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:55.854274 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:55.854564 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:07:55.855202 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:07:55.855207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:07:55.855211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:07:55.856048 (+ 837us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:07:55.856238 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 03:07:55.857596 (+ 1358us) server_negotiation.cc:299] Negotiation successful
0928 03:07:55.858136 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2686,"thread_start_us":2493,"threads_started":1}
I20260928 03:07:55.858610 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:55.830627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:35991 (local address 127.6.192.1:51271)
0928 03:07:55.834395 (+ 3768us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:55.834411 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:55.834560 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:55.836876 (+ 2316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:55.836881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:55.837735 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:55.838640 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:55.838668 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:55.840443 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:55.840447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:55.842003 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:55.842016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:55.843095 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:55.843103 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:55.843339 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:55.844364 (+ 1025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:55.844389 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:55.847288 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:55.852913 (+ 5625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:55.852925 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:55.852946 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:55.854012 (+ 1066us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:55.854839 (+ 827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:55.854847 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:55.854854 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:55.855069 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:55.856259 (+ 1190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:55.856269 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:55.856889 (+ 620us) client_negotiation.cc:769] Sending connection context
0928 03:07:55.857311 (+ 422us) client_negotiation.cc:240] Negotiation successful
0928 03:07:55.858024 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3619,"mutex_wait_us":425,"spinlock_wait_cycles":40960,"thread_start_us":3343,"threads_started":1}
I20260928 03:07:55.858603 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:55.830620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:32965 (local address 127.6.192.1:50415)
0928 03:07:55.831706 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:55.831769 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 03:07:55.832683 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:07:55.836793 (+ 4110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:07:55.836805 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:07:55.837728 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:07:55.838632 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:55.838665 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:55.840507 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:55.840512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:55.842002 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:07:55.842015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:07:55.843035 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:07:55.843042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:07:55.843342 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:07:55.844320 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:07:55.844352 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:07:55.847580 (+ 3228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:07:55.852913 (+ 5333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:55.852925 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:55.852946 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:55.854012 (+ 1066us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:55.854739 (+ 727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:07:55.854746 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:07:55.854750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:07:55.855069 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:07:55.856260 (+ 1191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:07:55.856269 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:07:55.856889 (+ 620us) client_negotiation.cc:769] Sending connection context
0928 03:07:55.857311 (+ 422us) client_negotiation.cc:240] Negotiation successful
0928 03:07:55.858182 (+ 871us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":439,"spinlock_wait_cycles":12416}
W20260928 03:07:55.862839 22000 leader_election.cc:341] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9db0158311340769f8387143a7a36ca (127.6.192.3:32965): Not found: Tablet not found: f7f84861679449cfbe0cdd43a7a6e523
W20260928 03:07:55.863183 21999 leader_election.cc:341] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5eaf1fab05534fd9b035ec488cec2486 (127.6.192.2:35991): Not found: Tablet not found: f7f84861679449cfbe0cdd43a7a6e523
I20260928 03:07:55.863370 21999 leader_election.cc:302] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82d60814dc2b49c7a7508d75cf039d18; no voters: 5eaf1fab05534fd9b035ec488cec2486, d9db0158311340769f8387143a7a36ca
I20260928 03:07:55.863967 22380 raft_consensus.cc:2726] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:07:56.028676 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:56.027606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:51069)
0928 03:07:56.028171 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:56.028323 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":284,"threads_started":1}
I20260928 03:07:56.496541 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:56.495708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:46229)
0928 03:07:56.496194 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:56.496320 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":233,"threads_started":1}
I20260928 03:07:56.523025 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:56.522248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:37037)
0928 03:07:56.522755 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:56.522856 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":276,"threads_started":1}
I20260928 03:07:57.030566 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:57.029854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:45345)
0928 03:07:57.030306 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:57.030392 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":226,"threads_started":1}
I20260928 03:07:57.498448 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:57.497596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:54363)
0928 03:07:57.498107 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:57.498201 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":274,"threads_started":1}
I20260928 03:07:57.525346 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:57.524211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:53873)
0928 03:07:57.524930 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:57.525066 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":336,"threads_started":1}
I20260928 03:07:57.929036 22390 raft_consensus.cc:482] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:07:57.929224 22390 raft_consensus.cc:504] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:57.929816 22390 leader_election.cc:288] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9db0158311340769f8387143a7a36ca (127.6.192.3:32965), 5eaf1fab05534fd9b035ec488cec2486 (127.6.192.2:35991)
W20260928 03:07:57.931145 22000 leader_election.cc:341] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9db0158311340769f8387143a7a36ca (127.6.192.3:32965): Not found: Tablet not found: f7f84861679449cfbe0cdd43a7a6e523
W20260928 03:07:57.931391 21999 leader_election.cc:341] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5eaf1fab05534fd9b035ec488cec2486 (127.6.192.2:35991): Not found: Tablet not found: f7f84861679449cfbe0cdd43a7a6e523
I20260928 03:07:57.931558 21999 leader_election.cc:302] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82d60814dc2b49c7a7508d75cf039d18; no voters: 5eaf1fab05534fd9b035ec488cec2486, d9db0158311340769f8387143a7a36ca
I20260928 03:07:57.931811 22390 raft_consensus.cc:2726] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:07:58.032442 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:58.031635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:59775)
0928 03:07:58.032130 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:58.032217 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":280,"threads_started":1}
I20260928 03:07:58.500293 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:58.499572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:44141)
0928 03:07:58.500040 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:58.500132 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":246,"threads_started":1}
I20260928 03:07:58.527220 22393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:58.526462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:35609)
0928 03:07:58.526947 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:58.527039 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":221,"threads_started":1}
I20260928 03:07:59.034616 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:59.033517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:56971)
0928 03:07:59.034301 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:59.034423 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":487,"threads_started":1}
I20260928 03:07:59.502380 22395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:59.501449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:48621)
0928 03:07:59.502070 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:59.502166 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":312,"threads_started":1}
I20260928 03:07:59.529152 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:07:59.528387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:55019)
0928 03:07:59.528860 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 03:07:59.528962 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":222,"threads_started":1}
I20260928 03:07:59.717789 22397 raft_consensus.cc:482] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:07:59.718065 22397 raft_consensus.cc:504] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:07:59.718935 22397 leader_election.cc:288] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9db0158311340769f8387143a7a36ca (127.6.192.3:32965), 5eaf1fab05534fd9b035ec488cec2486 (127.6.192.2:35991)
W20260928 03:07:59.719888 22000 leader_election.cc:341] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9db0158311340769f8387143a7a36ca (127.6.192.3:32965): Not found: Tablet not found: f7f84861679449cfbe0cdd43a7a6e523
W20260928 03:07:59.720104 21999 leader_election.cc:341] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5eaf1fab05534fd9b035ec488cec2486 (127.6.192.2:35991): Not found: Tablet not found: f7f84861679449cfbe0cdd43a7a6e523
I20260928 03:07:59.720239 21999 leader_election.cc:302] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82d60814dc2b49c7a7508d75cf039d18; no voters: 5eaf1fab05534fd9b035ec488cec2486, d9db0158311340769f8387143a7a36ca
I20260928 03:07:59.720505 22397 raft_consensus.cc:2726] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:08:00.036695 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:00.035869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:56647)
0928 03:08:00.036429 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:00.036525 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":288,"threads_started":1}
I20260928 03:08:00.504455 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:00.503647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:41789)
0928 03:08:00.504159 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:00.504255 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":276,"threads_started":1}
I20260928 03:08:00.531191 22400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:00.530315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:42401)
0928 03:08:00.530861 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:00.530991 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":307,"threads_started":1}
I20260928 03:08:01.038734 22401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:01.037851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:58735)
0928 03:08:01.038420 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:01.038522 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":279,"threads_started":1}
I20260928 03:08:01.506392 22402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:01.505618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:60149)
0928 03:08:01.506108 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:01.506195 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":271,"threads_started":1}
I20260928 03:08:01.533002 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:01.532226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:46501)
0928 03:08:01.532708 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:01.532797 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":265,"threads_started":1}
I20260928 03:08:02.041101 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:02.040053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:44903)
0928 03:08:02.040702 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:02.040884 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":290,"threads_started":1}
I20260928 03:08:02.418056 22405 raft_consensus.cc:482] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:08:02.418327 22405 raft_consensus.cc:504] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:08:02.419097 22405 leader_election.cc:288] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9db0158311340769f8387143a7a36ca (127.6.192.3:32965), 5eaf1fab05534fd9b035ec488cec2486 (127.6.192.2:35991)
W20260928 03:08:02.420121 22000 leader_election.cc:341] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9db0158311340769f8387143a7a36ca (127.6.192.3:32965): Not found: Tablet not found: f7f84861679449cfbe0cdd43a7a6e523
W20260928 03:08:02.420324 21999 leader_election.cc:341] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5eaf1fab05534fd9b035ec488cec2486 (127.6.192.2:35991): Not found: Tablet not found: f7f84861679449cfbe0cdd43a7a6e523
I20260928 03:08:02.420445 21999 leader_election.cc:302] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82d60814dc2b49c7a7508d75cf039d18; no voters: 5eaf1fab05534fd9b035ec488cec2486, d9db0158311340769f8387143a7a36ca
I20260928 03:08:02.420686 22405 raft_consensus.cc:2726] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:08:02.508371 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:02.507499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:55525)
0928 03:08:02.508072 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:02.508161 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":287,"threads_started":1}
I20260928 03:08:02.534924 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:02.533981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:40715)
0928 03:08:02.534564 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:02.534693 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":284,"threads_started":1}
I20260928 03:08:03.043011 22408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:03.042120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:38571)
0928 03:08:03.042650 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:03.042742 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":279,"threads_started":1}
I20260928 03:08:03.510169 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:03.509452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:49189)
0928 03:08:03.509908 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:03.509992 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":242,"threads_started":1}
I20260928 03:08:03.536809 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:03.536026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:44923)
0928 03:08:03.536514 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:03.536602 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":243,"threads_started":1}
I20260928 03:08:04.044728 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:04.043974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:56773)
0928 03:08:04.044477 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:04.044568 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":266,"threads_started":1}
I20260928 03:08:04.512230 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:04.511307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:53303)
0928 03:08:04.511893 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:04.512021 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":284,"threads_started":1}
I20260928 03:08:04.538995 22413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:04.538040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:53945)
0928 03:08:04.538734 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:04.538825 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":460,"threads_started":1}
I20260928 03:08:05.046800 22414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:05.045835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:49791)
0928 03:08:05.046400 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:05.046502 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":279,"threads_started":1}
I20260928 03:08:05.413604 22415 raft_consensus.cc:482] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:08:05.413878 22415 raft_consensus.cc:504] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9db0158311340769f8387143a7a36ca" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 32965 } } peers { permanent_uuid: "5eaf1fab05534fd9b035ec488cec2486" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35991 } } peers { permanent_uuid: "82d60814dc2b49c7a7508d75cf039d18" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 38953 } }
I20260928 03:08:05.414595 22415 leader_election.cc:288] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9db0158311340769f8387143a7a36ca (127.6.192.3:32965), 5eaf1fab05534fd9b035ec488cec2486 (127.6.192.2:35991)
W20260928 03:08:05.415831 22000 leader_election.cc:341] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9db0158311340769f8387143a7a36ca (127.6.192.3:32965): Not found: Tablet not found: f7f84861679449cfbe0cdd43a7a6e523
W20260928 03:08:05.416113 21999 leader_election.cc:341] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5eaf1fab05534fd9b035ec488cec2486 (127.6.192.2:35991): Not found: Tablet not found: f7f84861679449cfbe0cdd43a7a6e523
I20260928 03:08:05.416235 21999 leader_election.cc:302] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82d60814dc2b49c7a7508d75cf039d18; no voters: 5eaf1fab05534fd9b035ec488cec2486, d9db0158311340769f8387143a7a36ca
I20260928 03:08:05.416565 22415 raft_consensus.cc:2726] T f7f84861679449cfbe0cdd43a7a6e523 P 82d60814dc2b49c7a7508d75cf039d18 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:08:05.514003 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:05.513263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:54647)
0928 03:08:05.513716 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:05.513822 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":241,"threads_started":1}
I20260928 03:08:05.540874 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:05.539984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:57277)
0928 03:08:05.540506 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:05.540619 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":250,"threads_started":1}
I20260928 03:08:06.048918 22418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:06.048037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:34031)
0928 03:08:06.048568 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:06.048662 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":271,"threads_started":1}
I20260928 03:08:06.515820 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:06.515006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:56477)
0928 03:08:06.515573 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:06.515662 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":238,"threads_started":1}
I20260928 03:08:06.542896 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:06.541980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:36681)
0928 03:08:06.542593 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:06.542717 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":359,"threads_started":1}
I20260928 03:08:07.050647 22421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:07.049916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:35719)
0928 03:08:07.050384 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:07.050467 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":237,"threads_started":1}
I20260928 03:08:07.517669 22422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:07.516836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:46401)
0928 03:08:07.517393 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:07.517478 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":297,"threads_started":1}
I20260928 03:08:07.544866 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:07.544082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:44707)
0928 03:08:07.544590 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:07.544688 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":268,"threads_started":1}
I20260928 03:08:08.052845 22424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:08.051823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:54249)
0928 03:08:08.052477 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:08.052609 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":298,"threads_started":1}
I20260928 03:08:08.520159 22425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:08.519085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:43751)
0928 03:08:08.519836 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:08.519967 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":425,"threads_started":1}
I20260928 03:08:08.546722 22426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:08.545906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:34207)
0928 03:08:08.546425 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:08.546539 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":281,"threads_started":1}
I20260928 03:08:09.054857 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:09.053982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:40397)
0928 03:08:09.054538 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:09.054637 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":307,"threads_started":1}
I20260928 03:08:09.522055 22428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:09.521092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:57313)
0928 03:08:09.521628 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:09.521819 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":302,"threads_started":1}
I20260928 03:08:09.548851 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:09.548117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:50109)
0928 03:08:09.548588 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:09.548678 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":239,"threads_started":1}
I20260928 03:08:10.057173 22430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:10.056204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.2:57323)
0928 03:08:10.056912 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:10.057014 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":437,"threads_started":1}
I20260928 03:08:10.506417 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 21946
I20260928 03:08:10.524293 22432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:10.523404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.3:36667)
0928 03:08:10.524006 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:10.524123 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":287,"threads_started":1}
I20260928 03:08:10.530694 22431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:10.507974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59002 (local address 127.6.192.1:38953)
0928 03:08:10.508716 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:10.508736 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:10.508770 (+ 34us) server_negotiation.cc:407] Connection header received
0928 03:08:10.508841 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:10.508849 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:10.509145 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:10.509367 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:08:10.511255 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:10.512537 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:10.514118 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:10.514797 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:10.514976 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:10.523802 (+ 8826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:08:10.523830 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:08:10.523858 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:08:10.523900 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:08:10.527221 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:10.528231 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:10.528239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:10.528241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:10.528406 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:10.528750 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:10.528753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:10.528755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:10.529154 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:08:10.529491 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:10.530001 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 03:08:10.530320 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":331,"threads_started":1}
I20260928 03:08:10.532109 22106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:08:10.623570 22434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:10.622412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:43117 (local address 127.6.192.1:59465)
0928 03:08:10.622960 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:10.623119 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":268,"threads_started":1}
I20260928 03:08:10.623600 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 21946
I20260928 03:08:10.642153 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 22115
I20260928 03:08:10.658254 22435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:10.643052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.6.192.2:35991)
0928 03:08:10.643675 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:10.643681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:10.643699 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:08:10.643764 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:10.643769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:10.643866 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:10.643974 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:08:10.645371 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:10.646455 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:10.648292 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:10.648734 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:10.648848 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:10.652040 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:08:10.652045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:08:10.652048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:08:10.652088 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:08:10.654973 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:10.656183 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:10.656189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:10.656191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:10.656334 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:10.656728 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:10.656732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:10.656734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:10.657039 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:08:10.657319 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:10.657623 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 03:08:10.657816 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":301,"threads_started":1}
I20260928 03:08:10.659045 22238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:08:10.749042 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 22115
I20260928 03:08:10.767072 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 22247
I20260928 03:08:10.782701 22437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:10.768094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57768 (local address 127.6.192.3:32965)
0928 03:08:10.768730 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:10.768734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:10.768751 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:08:10.768805 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:10.768809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:10.768876 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:10.768981 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:08:10.770539 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:10.771572 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:10.773078 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:10.773507 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:10.773641 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:10.777049 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:08:10.777054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:08:10.777057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:08:10.777097 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:08:10.779870 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:10.780576 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:10.780580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:10.780582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:10.780722 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:10.781302 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:10.781305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:10.781307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:10.781556 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:08:10.781775 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:10.782132 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 03:08:10.782303 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":351,"threads_started":1}
I20260928 03:08:10.783576 22370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:08:10.876742 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 22247
2026-09-28T03:08:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25059 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:08:10 dist-test-slave-fjww krb5kdc[22441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:08:10 dist-test-slave-fjww krb5kdc[22441](info): set up 2 sockets
Sep 28 03:08:10 dist-test-slave-fjww krb5kdc[22441](info): commencing operation
krb5kdc: starting...
W20260928 03:08:12.017035 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:08:12 dist-test-slave-fjww krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:08:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:08:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:08:12.292382 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:36919
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:36919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:08:12.637653 22454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:08:12.638024 22454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:08:12.638197 22454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:08:12.645728 22454 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 03:08:12.645881 22454 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 03:08:12.645973 22454 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 03:08:12.646075 22454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:08:12.646166 22454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:08:12.646291 22454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:08:12.646381 22454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:08:12.646462 22454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:08:12.656819 22454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:36919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:36919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:08:12.658274 22454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:08:12.659823 22454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:08:12.674013 22459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:12.673974 22460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:12.673910 22462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:08:12.674522 22454 server_base.cc:1034] running on GCE node
I20260928 03:08:12.675431 22454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:08:12.677233 22454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:08:12.678498 22454 hybrid_clock.cc:648] HybridClock initialized: now 1790564892678412 us; error 71 us; skew 500 ppm
I20260928 03:08:12.678860 22454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:08:12 dist-test-slave-fjww krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564892, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:08:12.682765 22454 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:08:12.684542 22454 webserver.cc:460] Webserver started at http://127.6.192.62:39445/ using document root <none> and password file <none>
I20260928 03:08:12.685294 22454 fs_manager.cc:359] Metadata directory not provided
I20260928 03:08:12.685387 22454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:08:12.685655 22454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:08:12.688925 22454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "85cfa81497414912b54b17cc1fbfb2f1"
format_stamp: "Formatted at 2026-09-28 03:08:12 on dist-test-slave-fjww"
server_key: "86319ea8bf0de458565f6cd72d14daee"
server_key_iv: "38e750cf9ea6ec0c044bd9bf414369ff"
server_key_version: "encryptionkey@0"
I20260928 03:08:12.689690 22454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "85cfa81497414912b54b17cc1fbfb2f1"
format_stamp: "Formatted at 2026-09-28 03:08:12 on dist-test-slave-fjww"
server_key: "86319ea8bf0de458565f6cd72d14daee"
server_key_iv: "38e750cf9ea6ec0c044bd9bf414369ff"
server_key_version: "encryptionkey@0"
I20260928 03:08:12.695746 22454 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 03:08:12.699781 22469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:12.701468 22454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:08:12.701634 22454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "85cfa81497414912b54b17cc1fbfb2f1"
format_stamp: "Formatted at 2026-09-28 03:08:12 on dist-test-slave-fjww"
server_key: "86319ea8bf0de458565f6cd72d14daee"
server_key_iv: "38e750cf9ea6ec0c044bd9bf414369ff"
server_key_version: "encryptionkey@0"
I20260928 03:08:12.701838 22454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:08:12.725010 22454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:08:12.729838 22454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:08:12.730094 22454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:08:12.750617 22454 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:36919
I20260928 03:08:12.750655 22520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:36919 every 8 connection(s)
I20260928 03:08:12.752143 22454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:08:12.756167 22521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:08:12.758656 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 22454
I20260928 03:08:12.758862 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:08:12.759609 6912 external_mini_cluster.cc:1428] Setting key ac1bb4829527ce727c7546fd073ef0c4
I20260928 03:08:12.768975 22521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1: Bootstrap starting.
I20260928 03:08:12.773316 22521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1: Neither blocks nor log segments found. Creating new log.
I20260928 03:08:12.774783 22521 log.cc:824] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1: Log is configured to *not* fsync() on all Append() calls
Sep 28 03:08:12 dist-test-slave-fjww krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:08:12.778532 22521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1: No bootstrap required, opened a new log
I20260928 03:08:12.786811 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:12.761854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38394 (local address 127.6.192.62:36919)
0928 03:08:12.762718 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:12.762730 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:12.762773 (+ 43us) server_negotiation.cc:407] Connection header received
0928 03:08:12.764532 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:12.764556 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:12.764994 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:12.765346 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:08:12.767272 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:12.768678 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:12.770657 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:12.771379 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:12.771623 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:12.777061 (+ 5438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:08:12.777075 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:08:12.777098 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:08:12.777144 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:08:12.781357 (+ 4213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:12.782305 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:12.782315 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:12.782325 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:12.782554 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:12.783022 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:12.783033 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:12.783038 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:12.783820 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:08:12.784162 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:12.784737 (+ 575us) server_negotiation.cc:299] Negotiation successful
0928 03:08:12.785270 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":394,"threads_started":1}
I20260928 03:08:12.789511 22521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cfa81497414912b54b17cc1fbfb2f1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36919 } }
I20260928 03:08:12.789928 22521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:08:12.790066 22521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85cfa81497414912b54b17cc1fbfb2f1, State: Initialized, Role: FOLLOWER
I20260928 03:08:12.790714 22521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cfa81497414912b54b17cc1fbfb2f1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36919 } }
I20260928 03:08:12.791164 22521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:08:12.791335 22521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:08:12.791571 22521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:08:12.794031 22521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cfa81497414912b54b17cc1fbfb2f1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36919 } }
I20260928 03:08:12.794840 22521 leader_election.cc:302] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85cfa81497414912b54b17cc1fbfb2f1; no voters:
I20260928 03:08:12.795567 22521 leader_election.cc:288] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:08:12.795751 22526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:08:12.798223 22521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:08:12.801898 22526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 1 LEADER]: Becoming Leader. State: Replica: 85cfa81497414912b54b17cc1fbfb2f1, State: Running, Role: LEADER
I20260928 03:08:12.802884 22526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cfa81497414912b54b17cc1fbfb2f1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36919 } }
I20260928 03:08:12.806608 22527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85cfa81497414912b54b17cc1fbfb2f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cfa81497414912b54b17cc1fbfb2f1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36919 } } }
I20260928 03:08:12.807049 22527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [sys.catalog]: This master's current role is: LEADER
I20260928 03:08:12.808364 22528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85cfa81497414912b54b17cc1fbfb2f1. Latest consensus state: current_term: 1 leader_uuid: "85cfa81497414912b54b17cc1fbfb2f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cfa81497414912b54b17cc1fbfb2f1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36919 } } }
I20260928 03:08:12.808749 22528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [sys.catalog]: This master's current role is: LEADER
I20260928 03:08:12.812501 22532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:08:12.818260 22532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:08:12.828536 22532 catalog_manager.cc:1347] Generated new cluster ID: 4d92b99f608e4b9f8258e053e76b5350
I20260928 03:08:12.828684 22532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:08:12.852239 22532 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:08:12.854300 22532 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:08:12.861073 22532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1: Generated new TSK 0
I20260928 03:08:12.861939 22532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
I20260928 03:08:12.928429 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:36919
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:08:13.282269 22547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:08:13.282574 22547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:08:13.282740 22547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:08:13.290527 22547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:08:13.290666 22547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:08:13.290864 22547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:08:13.301817 22547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:36919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:08:13.303251 22547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:08:13.304745 22547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:08:13.319336 22555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:13.319237 22552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:13.319361 22553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:08:13.320040 22547 server_base.cc:1034] running on GCE node
I20260928 03:08:13.320756 22547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:08:13.321817 22547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:08:13.323127 22547 hybrid_clock.cc:648] HybridClock initialized: now 1790564893323049 us; error 64 us; skew 500 ppm
I20260928 03:08:13.323627 22547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:08:13 dist-test-slave-fjww krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564893, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:08:13.327533 22547 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:08:13.329409 22547 webserver.cc:460] Webserver started at http://127.6.192.1:33141/ using document root <none> and password file <none>
I20260928 03:08:13.330219 22547 fs_manager.cc:359] Metadata directory not provided
I20260928 03:08:13.330387 22547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:08:13.330776 22547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:08:13.334156 22547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "34c53cb0f3404a6aabaa805c6ada655a"
format_stamp: "Formatted at 2026-09-28 03:08:13 on dist-test-slave-fjww"
server_key: "94755972b76708e2073e0a78b7cd4bb9"
server_key_iv: "4da2e161491ca25c55b2003915fffc43"
server_key_version: "encryptionkey@0"
I20260928 03:08:13.335048 22547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "34c53cb0f3404a6aabaa805c6ada655a"
format_stamp: "Formatted at 2026-09-28 03:08:13 on dist-test-slave-fjww"
server_key: "94755972b76708e2073e0a78b7cd4bb9"
server_key_iv: "4da2e161491ca25c55b2003915fffc43"
server_key_version: "encryptionkey@0"
I20260928 03:08:13.340935 22547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 03:08:13.345009 22562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:13.346580 22547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:08:13.346803 22547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "34c53cb0f3404a6aabaa805c6ada655a"
format_stamp: "Formatted at 2026-09-28 03:08:13 on dist-test-slave-fjww"
server_key: "94755972b76708e2073e0a78b7cd4bb9"
server_key_iv: "4da2e161491ca25c55b2003915fffc43"
server_key_version: "encryptionkey@0"
I20260928 03:08:13.347060 22547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:08:13.378336 22547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:08:13.383181 22547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:08:13.383600 22547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:08:13.384696 22547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:08:13.386240 22547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:08:13.386317 22547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:13.386404 22547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:08:13.386447 22547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:13.421646 22547 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:37375
I20260928 03:08:13.421686 22674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:37375 every 8 connection(s)
I20260928 03:08:13.423061 22547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:08:13.428086 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 22547
I20260928 03:08:13.428310 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:08:13.428990 6912 external_mini_cluster.cc:1428] Setting key be5f73589d4d22c82d1420529de76193
Sep 28 03:08:13 dist-test-slave-fjww krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564893, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:08:13.445052 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:13.426077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:45047 (local address 127.6.192.62:36919)
0928 03:08:13.426523 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:13.426528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:13.427507 (+ 979us) server_negotiation.cc:407] Connection header received
0928 03:08:13.429763 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:13.429771 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:13.429860 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:13.429960 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:08:13.431832 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:13.432584 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:13.434284 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:13.434713 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:13.434859 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:13.438636 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:08:13.438642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:08:13.438645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:08:13.438681 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:08:13.441477 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:13.442520 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:13.442524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:13.442526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:13.442652 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:13.443173 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:13.443176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:13.443178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:13.443509 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:08:13.443628 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:13.444599 (+ 971us) server_negotiation.cc:299] Negotiation successful
0928 03:08:13.444785 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":222,"threads_started":1}
I20260928 03:08:13.446762 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:13.426557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:45047)
0928 03:08:13.427303 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:13.427354 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 03:08:13.429463 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:08:13.430236 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:08:13.430247 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:08:13.430925 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:08:13.431614 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:13.431635 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:13.432780 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:13.432785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:13.434103 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:13.434117 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:13.434978 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:13.434986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:13.435232 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:13.436038 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:08:13.436063 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:08:13.438373 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:08:13.441643 (+ 3270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:08:13.441656 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:08:13.441678 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:08:13.442368 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:08:13.442771 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:08:13.442779 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:08:13.442786 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:08:13.443054 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:08:13.443638 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:08:13.443649 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:08:13.444170 (+ 521us) client_negotiation.cc:769] Sending connection context
0928 03:08:13.444497 (+ 327us) client_negotiation.cc:240] Negotiation successful
0928 03:08:13.445118 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":311,"threads_started":1}
I20260928 03:08:13.448527 22675 heartbeater.cc:347] Connected to a master server at 127.6.192.62:36919
I20260928 03:08:13.448863 22675 heartbeater.cc:464] Registering TS with master...
I20260928 03:08:13.449648 22675 heartbeater.cc:511] Master 127.6.192.62:36919 requested a full tablet report, sending...
I20260928 03:08:13.452129 22486 ts_manager.cc:194] Registered new tserver with Master: 34c53cb0f3404a6aabaa805c6ada655a (127.6.192.1:37375)
WARNING: no policy specified for kudu/127.6.192.2@KRBTEST.COM; defaulting to no policy
I20260928 03:08:13.453665 22486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:45047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
I20260928 03:08:13.488224 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:36919
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:08:13.826617 22681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:08:13.826922 22681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:08:13.827085 22681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:08:13.834522 22681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:08:13.834648 22681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:08:13.834851 22681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:08:13.844846 22681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:36919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:08:13.846207 22681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:08:13.847682 22681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:08:13.861547 22686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:13.861716 22687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:13.862135 22689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:08:13.865319 22681 server_base.cc:1034] running on GCE node
I20260928 03:08:13.866070 22681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:08:13.867141 22681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:08:13.868438 22681 hybrid_clock.cc:648] HybridClock initialized: now 1790564893868374 us; error 58 us; skew 500 ppm
I20260928 03:08:13.868985 22681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:08:13 dist-test-slave-fjww krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564893, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:08:13.873171 22681 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:08:13.875124 22681 webserver.cc:460] Webserver started at http://127.6.192.2:38927/ using document root <none> and password file <none>
I20260928 03:08:13.876029 22681 fs_manager.cc:359] Metadata directory not provided
I20260928 03:08:13.876156 22681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:08:13.876488 22681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:08:13.880362 22681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "4322c37b370b4da3afb2b962da7f94d3"
format_stamp: "Formatted at 2026-09-28 03:08:13 on dist-test-slave-fjww"
server_key: "e971b55baac6676975eb606409ef1376"
server_key_iv: "3c7b117086cff6806ac5ca2cfa6f9a88"
server_key_version: "encryptionkey@0"
I20260928 03:08:13.881291 22681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "4322c37b370b4da3afb2b962da7f94d3"
format_stamp: "Formatted at 2026-09-28 03:08:13 on dist-test-slave-fjww"
server_key: "e971b55baac6676975eb606409ef1376"
server_key_iv: "3c7b117086cff6806ac5ca2cfa6f9a88"
server_key_version: "encryptionkey@0"
I20260928 03:08:13.887115 22681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 03:08:13.890861 22696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:13.892655 22681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:08:13.892840 22681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "4322c37b370b4da3afb2b962da7f94d3"
format_stamp: "Formatted at 2026-09-28 03:08:13 on dist-test-slave-fjww"
server_key: "e971b55baac6676975eb606409ef1376"
server_key_iv: "3c7b117086cff6806ac5ca2cfa6f9a88"
server_key_version: "encryptionkey@0"
I20260928 03:08:13.893059 22681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:08:13.909296 22681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:08:13.914212 22681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:08:13.914515 22681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:08:13.915596 22681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:08:13.917068 22681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:08:13.917142 22681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:13.917284 22681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:08:13.917336 22681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:13.953152 22681 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:36955
I20260928 03:08:13.953243 22808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:36955 every 8 connection(s)
I20260928 03:08:13.954500 22681 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:08:13.963862 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 22681
I20260928 03:08:13.964069 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:08:13.964581 6912 external_mini_cluster.cc:1428] Setting key c35b9f7180ec4d435fc14a4e23c5395c
Sep 28 03:08:13 dist-test-slave-fjww krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564893, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:08:13.978565 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:13.957214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:53087 (local address 127.6.192.62:36919)
0928 03:08:13.957666 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:13.957670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:13.958726 (+ 1056us) server_negotiation.cc:407] Connection header received
0928 03:08:13.960744 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:13.960749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:13.960859 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:13.960983 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:08:13.962623 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:13.963356 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:13.965232 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:13.965739 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:13.965852 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:13.970516 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:08:13.970526 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:08:13.970534 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:08:13.970584 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:08:13.974545 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:13.975611 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:13.975620 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:13.975625 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:13.975794 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:13.976454 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:13.976458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:13.976460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:13.976733 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:08:13.976848 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:13.977756 (+ 908us) server_negotiation.cc:299] Negotiation successful
0928 03:08:13.978053 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":208,"threads_started":1}
I20260928 03:08:13.979202 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:13.957801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:53087)
0928 03:08:13.958535 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:13.958581 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 03:08:13.960454 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:08:13.961230 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:08:13.961239 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:08:13.961796 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:08:13.962435 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:13.962454 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:13.963557 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:13.963561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:13.965036 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:13.965051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:13.966003 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:13.966010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:13.966328 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:13.967220 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:08:13.967296 (+ 76us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:08:13.970192 (+ 2896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:08:13.974777 (+ 4585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:08:13.974861 (+ 84us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:08:13.974877 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:08:13.975403 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:08:13.975942 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:08:13.975949 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:08:13.975954 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:08:13.976273 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:08:13.976874 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:08:13.976888 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:08:13.977324 (+ 436us) client_negotiation.cc:769] Sending connection context
0928 03:08:13.977573 (+ 249us) client_negotiation.cc:240] Negotiation successful
0928 03:08:13.978005 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":287,"threads_started":1}
I20260928 03:08:13.980911 22809 heartbeater.cc:347] Connected to a master server at 127.6.192.62:36919
I20260928 03:08:13.981261 22809 heartbeater.cc:464] Registering TS with master...
I20260928 03:08:13.982054 22809 heartbeater.cc:511] Master 127.6.192.62:36919 requested a full tablet report, sending...
I20260928 03:08:13.983749 22486 ts_manager.cc:194] Registered new tserver with Master: 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955)
I20260928 03:08:13.984642 22486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:53087
WARNING: no policy specified for kudu/127.6.192.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
I20260928 03:08:14.024932 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:36919
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:08:14.367843 22815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:08:14.368151 22815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:08:14.368352 22815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:08:14.375912 22815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:08:14.376039 22815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:08:14.376231 22815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:08:14.395200 22815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:36919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:08:14.397079 22815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:08:14.398891 22815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:08:14.417112 22820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:14.417106 22821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:14.417176 22823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:08:14.418202 22815 server_base.cc:1034] running on GCE node
I20260928 03:08:14.419185 22815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:08:14.427688 22815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:08:14.429061 22815 hybrid_clock.cc:648] HybridClock initialized: now 1790564894428943 us; error 122 us; skew 500 ppm
I20260928 03:08:14.429513 22815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:08:14 dist-test-slave-fjww krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564894, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:08:14.433645 22815 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:08:14.435734 22815 webserver.cc:460] Webserver started at http://127.6.192.3:41749/ using document root <none> and password file <none>
I20260928 03:08:14.436771 22815 fs_manager.cc:359] Metadata directory not provided
I20260928 03:08:14.436924 22815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:08:14.437320 22815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:08:14.441454 22815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "7b85a9cf7cb14e0e9b734746e110f6f3"
format_stamp: "Formatted at 2026-09-28 03:08:14 on dist-test-slave-fjww"
server_key: "f47abc939473a2973891f34a30f12542"
server_key_iv: "af984c8d48215ba61c898215379ebcfe"
server_key_version: "encryptionkey@0"
I20260928 03:08:14.442278 22815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "7b85a9cf7cb14e0e9b734746e110f6f3"
format_stamp: "Formatted at 2026-09-28 03:08:14 on dist-test-slave-fjww"
server_key: "f47abc939473a2973891f34a30f12542"
server_key_iv: "af984c8d48215ba61c898215379ebcfe"
server_key_version: "encryptionkey@0"
I20260928 03:08:14.448905 22815 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 03:08:14.453752 22830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:14.456066 22815 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 03:08:14.456293 22815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "7b85a9cf7cb14e0e9b734746e110f6f3"
format_stamp: "Formatted at 2026-09-28 03:08:14 on dist-test-slave-fjww"
server_key: "f47abc939473a2973891f34a30f12542"
server_key_iv: "af984c8d48215ba61c898215379ebcfe"
server_key_version: "encryptionkey@0"
I20260928 03:08:14.456555 22815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:08:14.456830 22675 heartbeater.cc:503] Master 127.6.192.62:36919 was elected leader, sending a full tablet report...
I20260928 03:08:14.482242 22815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:08:14.487582 22815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:08:14.487860 22815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:08:14.488881 22815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:08:14.490238 22815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:08:14.490325 22815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:14.490413 22815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:08:14.490458 22815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:14.527405 22815 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:43711
I20260928 03:08:14.527540 22942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:43711 every 8 connection(s)
I20260928 03:08:14.528831 22815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:08:14.532827 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 22815
I20260928 03:08:14.533078 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:08:14.533674 6912 external_mini_cluster.cc:1428] Setting key de5096b9be5988bd12bbd9601adb0f68
Sep 28 03:08:14 dist-test-slave-fjww krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564894, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:08:14.552114 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:14.531746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:44739 (local address 127.6.192.62:36919)
0928 03:08:14.532292 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:14.532298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:14.533382 (+ 1084us) server_negotiation.cc:407] Connection header received
0928 03:08:14.536191 (+ 2809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:14.536205 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:14.536305 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:14.536435 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:08:14.538355 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.539177 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.540976 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.541430 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.541592 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:14.545412 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:08:14.545417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:08:14.545420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:08:14.545456 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:08:14.548414 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:14.549435 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:14.549439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:14.549441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:14.549572 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:14.550039 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:14.550043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:14.550045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:14.550371 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:08:14.550537 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:14.551418 (+ 881us) server_negotiation.cc:299] Negotiation successful
0928 03:08:14.551701 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":295,"threads_started":1}
I20260928 03:08:14.553115 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:14.532219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:44739)
0928 03:08:14.533202 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:14.533259 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 03:08:14.535685 (+ 2426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:08:14.536714 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:08:14.536723 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:08:14.537497 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:08:14.538156 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:14.538176 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.539374 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.539378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:14.540795 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:14.540810 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.541694 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.541712 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:14.541980 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:14.542727 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:08:14.542751 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:08:14.545157 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:08:14.548573 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:08:14.548583 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:08:14.548605 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:08:14.549285 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:08:14.549690 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:08:14.549695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:08:14.549699 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:08:14.549902 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:08:14.550508 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:08:14.550517 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:08:14.550997 (+ 480us) client_negotiation.cc:769] Sending connection context
0928 03:08:14.551289 (+ 292us) client_negotiation.cc:240] Negotiation successful
0928 03:08:14.551914 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":384,"threads_started":1}
I20260928 03:08:14.554973 22943 heartbeater.cc:347] Connected to a master server at 127.6.192.62:36919
I20260928 03:08:14.555356 22943 heartbeater.cc:464] Registering TS with master...
I20260928 03:08:14.556217 22943 heartbeater.cc:511] Master 127.6.192.62:36919 requested a full tablet report, sending...
I20260928 03:08:14.557943 22486 ts_manager.cc:194] Registered new tserver with Master: 7b85a9cf7cb14e0e9b734746e110f6f3 (127.6.192.3:43711)
I20260928 03:08:14.558938 22486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:44739
I20260928 03:08:14.563212 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:08:14.582988 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:14.568354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.6.192.62:36919)
0928 03:08:14.568564 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:14.568569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:14.569215 (+ 646us) server_negotiation.cc:407] Connection header received
0928 03:08:14.569429 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:14.569434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:14.569511 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:14.569610 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:08:14.571210 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.572119 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.573663 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.574118 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.574244 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:14.577089 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:08:14.577101 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:08:14.577105 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:08:14.577147 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:08:14.579958 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:14.580769 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:14.580774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:14.580777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:14.580914 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:14.581362 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:14.581368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:14.581370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:14.581871 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:08:14.582114 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:14.582441 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 03:08:14.582638 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 03:08:14.588223 22486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:08:14.595793 22486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:08:14.630897 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:14.609758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:43711 (local address 127.0.0.1:58778)
0928 03:08:14.611098 (+ 1340us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:14.611148 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 03:08:14.617483 (+ 6335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:08:14.622092 (+ 4609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:08:14.622101 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:08:14.622138 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:08:14.622744 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:14.622760 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.625159 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.625165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:14.627472 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:14.627493 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.629528 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.629535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:14.629786 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:14.629870 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 03:08:14.630002 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 03:08:14.630243 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1040,"mutex_wait_us":47,"spinlock_wait_cycles":5504,"thread_start_us":12022,"threads_started":1}
I20260928 03:08:14.631525 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:14.610644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50210 (local address 127.6.192.2:36955)
0928 03:08:14.620696 (+ 10052us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:14.620709 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:14.621386 (+ 677us) server_negotiation.cc:407] Connection header received
0928 03:08:14.621758 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:14.621773 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:14.622075 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:14.622297 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:08:14.622982 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.624714 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.627479 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.629251 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.629416 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:14.629664 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:14.630892 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0928 03:08:14.631095 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9877,"thread_start_us":9741,"threads_started":1}
I20260928 03:08:14.631669 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:14.610800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:37375 (local address 127.0.0.1:49788)
0928 03:08:14.622159 (+ 11359us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:14.622181 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 03:08:14.622360 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:08:14.623495 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:08:14.623500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:08:14.623545 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:08:14.624016 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:14.624032 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.626340 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.626345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:14.628435 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:14.628476 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.630752 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.630763 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:14.630962 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:14.630995 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 03:08:14.631072 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 03:08:14.631211 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11194,"mutex_wait_us":57,"spinlock_wait_cycles":3840,"thread_start_us":10293,"threads_started":1}
I20260928 03:08:14.631174 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:14.610072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:36955 (local address 127.0.0.1:50210)
0928 03:08:14.621240 (+ 11168us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:14.621263 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 03:08:14.621511 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:08:14.622279 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:08:14.622289 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:08:14.622367 (+ 78us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:08:14.622811 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:14.622822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.624873 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.624880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:14.627249 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:14.627264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.629835 (+ 2571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.629842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:14.630044 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:14.630074 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 03:08:14.630164 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 03:08:14.630283 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11020,"mutex_wait_us":31,"spinlock_wait_cycles":34432,"thread_start_us":749,"threads_started":1}
I20260928 03:08:14.632335 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:14.610059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58778 (local address 127.6.192.3:43711)
0928 03:08:14.621196 (+ 11137us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:14.621206 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:14.621252 (+ 46us) server_negotiation.cc:407] Connection header received
0928 03:08:14.621394 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:14.621405 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:14.621753 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:14.623091 (+ 1338us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:08:14.623422 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.625023 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.627658 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.629269 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.631303 (+ 2034us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:14.631623 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:14.631705 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 03:08:14.631907 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10944,"thread_start_us":10754,"threads_started":1}
I20260928 03:08:14.632624 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:14.611596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49788 (local address 127.6.192.1:37375)
0928 03:08:14.620305 (+ 8709us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:14.620319 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:14.622520 (+ 2201us) server_negotiation.cc:407] Connection header received
0928 03:08:14.622644 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:14.622656 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:14.622967 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:14.623180 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:08:14.624298 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.626177 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.628659 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.630078 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.630251 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:14.630480 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:14.631995 (+ 1515us) server_negotiation.cc:299] Negotiation successful
0928 03:08:14.632221 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8456,"thread_start_us":8256,"threads_started":1}
I20260928 03:08:14.636390 22878 tablet_service.cc:1467] Processing CreateTablet for tablet c71bb10b5a1c407ca1598799f79ba7ee (DEFAULT_TABLE table=test-table [id=5b540a5821684a308c4ff53e2402e354]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:08:14.636425 22744 tablet_service.cc:1467] Processing CreateTablet for tablet c71bb10b5a1c407ca1598799f79ba7ee (DEFAULT_TABLE table=test-table [id=5b540a5821684a308c4ff53e2402e354]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:08:14.636423 22610 tablet_service.cc:1467] Processing CreateTablet for tablet c71bb10b5a1c407ca1598799f79ba7ee (DEFAULT_TABLE table=test-table [id=5b540a5821684a308c4ff53e2402e354]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:08:14.637774 22878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c71bb10b5a1c407ca1598799f79ba7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:08:14.637923 22610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c71bb10b5a1c407ca1598799f79ba7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:08:14.637936 22744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c71bb10b5a1c407ca1598799f79ba7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:08:14.649756 22962 tablet_bootstrap.cc:492] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3: Bootstrap starting.
I20260928 03:08:14.654265 22962 tablet_bootstrap.cc:654] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3: Neither blocks nor log segments found. Creating new log.
I20260928 03:08:14.654978 22964 tablet_bootstrap.cc:492] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3: Bootstrap starting.
I20260928 03:08:14.655936 22962 log.cc:824] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3: Log is configured to *not* fsync() on all Append() calls
I20260928 03:08:14.659785 22964 tablet_bootstrap.cc:654] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3: Neither blocks nor log segments found. Creating new log.
I20260928 03:08:14.660367 22963 tablet_bootstrap.cc:492] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a: Bootstrap starting.
I20260928 03:08:14.661561 22964 log.cc:824] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3: Log is configured to *not* fsync() on all Append() calls
I20260928 03:08:14.664784 22963 tablet_bootstrap.cc:654] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a: Neither blocks nor log segments found. Creating new log.
I20260928 03:08:14.665733 22962 tablet_bootstrap.cc:492] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3: No bootstrap required, opened a new log
I20260928 03:08:14.666154 22962 ts_tablet_manager.cc:1397] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20260928 03:08:14.666299 22963 log.cc:824] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a: Log is configured to *not* fsync() on all Append() calls
I20260928 03:08:14.669504 22963 tablet_bootstrap.cc:492] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a: No bootstrap required, opened a new log
I20260928 03:08:14.669875 22963 ts_tablet_manager.cc:1397] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 03:08:14.674046 22964 tablet_bootstrap.cc:492] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3: No bootstrap required, opened a new log
I20260928 03:08:14.674372 22964 ts_tablet_manager.cc:1397] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.000s
I20260928 03:08:14.675590 22962 raft_consensus.cc:348] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } }
I20260928 03:08:14.676458 22962 raft_consensus.cc:374] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:08:14.676625 22962 raft_consensus.cc:729] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b85a9cf7cb14e0e9b734746e110f6f3, State: Initialized, Role: FOLLOWER
I20260928 03:08:14.677542 22962 consensus_queue.cc:260] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } }
I20260928 03:08:14.678833 22943 heartbeater.cc:503] Master 127.6.192.62:36919 was elected leader, sending a full tablet report...
I20260928 03:08:14.679675 22962 ts_tablet_manager.cc:1428] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260928 03:08:14.680824 22963 raft_consensus.cc:348] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } }
I20260928 03:08:14.681353 22963 raft_consensus.cc:374] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:08:14.681460 22963 raft_consensus.cc:729] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34c53cb0f3404a6aabaa805c6ada655a, State: Initialized, Role: FOLLOWER
I20260928 03:08:14.682125 22963 consensus_queue.cc:260] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } }
I20260928 03:08:14.684137 22963 ts_tablet_manager.cc:1428] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260928 03:08:14.685521 22964 raft_consensus.cc:348] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } }
I20260928 03:08:14.686266 22964 raft_consensus.cc:374] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:08:14.686388 22964 raft_consensus.cc:729] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4322c37b370b4da3afb2b962da7f94d3, State: Initialized, Role: FOLLOWER
I20260928 03:08:14.687105 22964 consensus_queue.cc:260] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } }
I20260928 03:08:14.688532 22809 heartbeater.cc:503] Master 127.6.192.62:36919 was elected leader, sending a full tablet report...
I20260928 03:08:14.689129 22964 ts_tablet_manager.cc:1428] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260928 03:08:14.706847 22810 tablet.cc:2369] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:08:14.720737 22968 raft_consensus.cc:482] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:08:14.721048 22968 raft_consensus.cc:504] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } }
I20260928 03:08:14.722906 22968 leader_election.cc:288] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955), 34c53cb0f3404a6aabaa805c6ada655a (127.6.192.1:37375)
I20260928 03:08:14.729003 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:14.723486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:41415 (local address 127.6.192.1:37375)
0928 03:08:14.723707 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:14.723713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:14.723737 (+ 24us) server_negotiation.cc:407] Connection header received
0928 03:08:14.723962 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:14.723966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:14.724032 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:14.724179 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:08:14.724670 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.725452 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.727156 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.728059 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.728213 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:14.728333 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:14.728609 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 03:08:14.728742 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 03:08:14.729035 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:14.723378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:37375 (local address 127.6.192.3:41415)
0928 03:08:14.723589 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:14.723618 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 03:08:14.723763 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:08:14.724219 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:08:14.724230 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:08:14.724274 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:08:14.724506 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:14.724515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.725608 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.725615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:14.726978 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:14.726991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.728231 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.728237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:14.728436 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:14.728461 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 03:08:14.728569 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 03:08:14.728716 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 03:08:14.730034 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:14.723539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:36383 (local address 127.6.192.2:36955)
0928 03:08:14.723747 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:14.723753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:14.724611 (+ 858us) server_negotiation.cc:407] Connection header received
0928 03:08:14.724797 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:14.724807 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:14.724867 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:14.724967 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:08:14.725498 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.726233 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.727994 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.728803 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.729010 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:14.729125 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:14.729670 (+ 545us) server_negotiation.cc:299] Negotiation successful
0928 03:08:14.729797 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 03:08:14.730186 22971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:14.723378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:36955 (local address 127.6.192.3:36383)
0928 03:08:14.724494 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:14.724509 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 03:08:14.724655 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:08:14.724985 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:08:14.724990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:08:14.725015 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:08:14.725342 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:14.725350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.726393 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.726397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:14.727829 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:14.727840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.729053 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.729059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:14.729262 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:14.729310 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 03:08:14.729628 (+ 318us) client_negotiation.cc:240] Negotiation successful
0928 03:08:14.729778 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"mutex_wait_us":272,"thread_start_us":480,"threads_started":1}
I20260928 03:08:14.730670 22630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c71bb10b5a1c407ca1598799f79ba7ee" candidate_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34c53cb0f3404a6aabaa805c6ada655a" is_pre_election: true
I20260928 03:08:14.731276 22630 raft_consensus.cc:2454] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b85a9cf7cb14e0e9b734746e110f6f3 in term 0.
I20260928 03:08:14.731570 22764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c71bb10b5a1c407ca1598799f79ba7ee" candidate_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4322c37b370b4da3afb2b962da7f94d3" is_pre_election: true
I20260928 03:08:14.732108 22831 leader_election.cc:302] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34c53cb0f3404a6aabaa805c6ada655a, 7b85a9cf7cb14e0e9b734746e110f6f3; no voters:
I20260928 03:08:14.732172 22764 raft_consensus.cc:2454] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b85a9cf7cb14e0e9b734746e110f6f3 in term 0.
I20260928 03:08:14.732573 22968 raft_consensus.cc:2781] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:08:14.732728 22968 raft_consensus.cc:482] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:08:14.732847 22968 raft_consensus.cc:3037] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:08:14.735195 22968 raft_consensus.cc:504] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } }
I20260928 03:08:14.735951 22968 leader_election.cc:288] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [CANDIDATE]: Term 1 election: Requested vote from peers 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955), 34c53cb0f3404a6aabaa805c6ada655a (127.6.192.1:37375)
I20260928 03:08:14.736490 22764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c71bb10b5a1c407ca1598799f79ba7ee" candidate_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4322c37b370b4da3afb2b962da7f94d3"
I20260928 03:08:14.736629 22630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c71bb10b5a1c407ca1598799f79ba7ee" candidate_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34c53cb0f3404a6aabaa805c6ada655a"
I20260928 03:08:14.736724 22764 raft_consensus.cc:3037] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:08:14.736923 22630 raft_consensus.cc:3037] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:08:14.739424 22764 raft_consensus.cc:2454] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b85a9cf7cb14e0e9b734746e110f6f3 in term 1.
I20260928 03:08:14.739673 22630 raft_consensus.cc:2454] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b85a9cf7cb14e0e9b734746e110f6f3 in term 1.
I20260928 03:08:14.739996 22834 leader_election.cc:302] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4322c37b370b4da3afb2b962da7f94d3, 7b85a9cf7cb14e0e9b734746e110f6f3; no voters:
I20260928 03:08:14.740367 22968 raft_consensus.cc:2781] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:08:14.740900 22968 raft_consensus.cc:686] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 1 LEADER]: Becoming Leader. State: Replica: 7b85a9cf7cb14e0e9b734746e110f6f3, State: Running, Role: LEADER
I20260928 03:08:14.741415 22968 consensus_queue.cc:237] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } }
I20260928 03:08:14.746752 22485 catalog_manager.cc:5495] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b85a9cf7cb14e0e9b734746e110f6f3 (127.6.192.3). New cstate: current_term: 1 leader_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } health_report { overall_health: UNKNOWN } } }
W20260928 03:08:14.780660 22944 tablet.cc:2369] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:08:14.782254 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:14.776339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58788 (local address 127.6.192.3:43711)
0928 03:08:14.776534 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:14.776538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:14.776556 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:08:14.776810 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:14.776814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:14.776906 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:14.777047 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:08:14.777485 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.778197 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.779801 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.780193 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.780288 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:14.781155 (+ 867us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:08:14.781730 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:08:14.781851 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:14.781975 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 03:08:14.782047 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 03:08:14.791620 22630 raft_consensus.cc:1261] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Refusing update from remote peer 7b85a9cf7cb14e0e9b734746e110f6f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:08:14.791613 22764 raft_consensus.cc:1261] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3 [term 1 FOLLOWER]: Refusing update from remote peer 7b85a9cf7cb14e0e9b734746e110f6f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:08:14.792783 22972 consensus_queue.cc:1035] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:08:14.793213 22968 consensus_queue.cc:1035] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:08:14.819132 22976 mvcc.cc:205] Tried to move back new op lower bound from 7334153809054097408 to 7334153808864428032. Current Snapshot: MvccSnapshot[applied={T|T < 7334153809054097408}]
I20260928 03:08:14.830571 22485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:08:14.830966 22485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:08:14.835831 22485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1: Sending DeleteTablet for 3 replicas of tablet c71bb10b5a1c407ca1598799f79ba7ee
I20260928 03:08:14.837375 22744 tablet_service.cc:1514] Processing DeleteTablet for tablet c71bb10b5a1c407ca1598799f79ba7ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:08:14 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:50210
I20260928 03:08:14.837423 22610 tablet_service.cc:1514] Processing DeleteTablet for tablet c71bb10b5a1c407ca1598799f79ba7ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:08:14 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:49788
I20260928 03:08:14.837587 22878 tablet_service.cc:1514] Processing DeleteTablet for tablet c71bb10b5a1c407ca1598799f79ba7ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:08:14 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:58778
I20260928 03:08:14.842089 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 22547
I20260928 03:08:14.847728 22986 tablet_replica.cc:330] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3: stopping tablet replica
I20260928 03:08:14.848438 22986 raft_consensus.cc:2229] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:08:14.849171 22985 tablet_replica.cc:330] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a: stopping tablet replica
I20260928 03:08:14.849967 22986 raft_consensus.cc:2258] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:08:14.850569 22985 raft_consensus.cc:2229] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:08:14.853219 22986 ts_tablet_manager.cc:1905] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:08:14.853848 22985 raft_consensus.cc:2258] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:08:14.855044 22984 tablet_replica.cc:330] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3: stopping tablet replica
I20260928 03:08:14.855521 22984 raft_consensus.cc:2229] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:08:14.856024 22984 raft_consensus.cc:2258] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:08:14.858289 22985 ts_tablet_manager.cc:1905] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:08:14.858281 22984 ts_tablet_manager.cc:1905] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:08:14.865703 22984 ts_tablet_manager.cc:1918] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:08:14.866070 22984 log.cc:1196] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/c71bb10b5a1c407ca1598799f79ba7ee
I20260928 03:08:14.866757 22984 ts_tablet_manager.cc:1939] T c71bb10b5a1c407ca1598799f79ba7ee P 4322c37b370b4da3afb2b962da7f94d3: Deleting consensus metadata
Sep 28 03:08:14 dist-test-slave-fjww krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:08:14.867676 22985 ts_tablet_manager.cc:1918] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:08:14.868978 22473 catalog_manager.cc:4841] TS 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955): tablet c71bb10b5a1c407ca1598799f79ba7ee (table test-table [id=5b540a5821684a308c4ff53e2402e354]) successfully deleted
I20260928 03:08:14.869673 22985 log.cc:1196] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/c71bb10b5a1c407ca1598799f79ba7ee
I20260928 03:08:14.870354 22985 ts_tablet_manager.cc:1939] T c71bb10b5a1c407ca1598799f79ba7ee P 34c53cb0f3404a6aabaa805c6ada655a: Deleting consensus metadata
I20260928 03:08:14.872531 22470 catalog_manager.cc:4841] TS 34c53cb0f3404a6aabaa805c6ada655a (127.6.192.1:37375): tablet c71bb10b5a1c407ca1598799f79ba7ee (table test-table [id=5b540a5821684a308c4ff53e2402e354]) successfully deleted
I20260928 03:08:14.873656 22986 ts_tablet_manager.cc:1918] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:08:14.873859 22986 log.cc:1196] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/c71bb10b5a1c407ca1598799f79ba7ee
I20260928 03:08:14.874429 22986 ts_tablet_manager.cc:1939] T c71bb10b5a1c407ca1598799f79ba7ee P 7b85a9cf7cb14e0e9b734746e110f6f3: Deleting consensus metadata
I20260928 03:08:14.876184 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:14.843934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.6.192.1:37375)
0928 03:08:14.844204 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:14.844214 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:14.848114 (+ 3900us) server_negotiation.cc:407] Connection header received
0928 03:08:14.854020 (+ 5906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:14.854031 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:14.854183 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:14.854283 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:08:14.856579 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.857775 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.860226 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:14.860887 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:14.860991 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:14.868581 (+ 7590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:08:14.868597 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:08:14.868615 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:08:14.868674 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:08:14.872843 (+ 4169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:14.873720 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:14.873732 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:14.873735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:14.873965 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:14.874386 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:14.874391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:14.874394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:14.874833 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:08:14.875148 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:14.875570 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 03:08:14.875804 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 03:08:14.876305 22471 catalog_manager.cc:4841] TS 7b85a9cf7cb14e0e9b734746e110f6f3 (127.6.192.3:43711): tablet c71bb10b5a1c407ca1598799f79ba7ee (table test-table [id=5b540a5821684a308c4ff53e2402e354]) successfully deleted
I20260928 03:08:14.877004 22670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:08:14.971180 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 22547
I20260928 03:08:14.997236 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 22681
Sep 28 03:08:15 dist-test-slave-fjww krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.2@KRBTEST.COM
I20260928 03:08:15.015504 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:14.998146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50214 (local address 127.6.192.2:36955)
0928 03:08:14.998394 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:14.998400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:14.998422 (+ 22us) server_negotiation.cc:407] Connection header received
0928 03:08:14.998592 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:14.998599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:14.998706 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:14.998825 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:08:15.000468 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:15.001354 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:15.002836 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:15.003244 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:15.003403 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:15.008475 (+ 5072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:08:15.008493 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:08:15.008510 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:08:15.008560 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:08:15.012143 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:15.013137 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:15.013147 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:15.013149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:15.013287 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:15.013718 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:15.013724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:15.013728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:15.014158 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:08:15.014473 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:15.014826 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 03:08:15.015075 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 03:08:15.016204 22804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:08:15.117249 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 22681
I20260928 03:08:15.134860 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 22815
Sep 28 03:08:15 dist-test-slave-fjww krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.3@KRBTEST.COM
I20260928 03:08:15.154734 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:15.135770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.6.192.3:43711)
0928 03:08:15.136038 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:15.136045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:15.136067 (+ 22us) server_negotiation.cc:407] Connection header received
0928 03:08:15.136141 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:15.136147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:15.136229 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:15.136380 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:08:15.137810 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:15.139163 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:15.140839 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:15.141521 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:15.141696 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:15.147123 (+ 5427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:08:15.147139 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:08:15.147159 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:08:15.147207 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:08:15.151099 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:15.152151 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:15.152160 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:15.152162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:15.152355 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:15.152849 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:15.152854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:15.152858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:15.153298 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:08:15.153547 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:15.154121 (+ 574us) server_negotiation.cc:299] Negotiation successful
0928 03:08:15.154299 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260928 03:08:15.155682 22938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:08:15.282279 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 22815
I20260928 03:08:15.303265 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 22454
I20260928 03:08:15.304523 22516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:08:15.417557 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 22454
I20260928 03:08:35.440162 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:36919
--webserver_interface=127.6.192.62
--webserver_port=39445
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:36919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:08:35.799746 22992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:08:35.800078 22992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:08:35.800287 22992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:08:35.808012 22992 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 03:08:35.808145 22992 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 03:08:35.808190 22992 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 03:08:35.808282 22992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:08:35.808370 22992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:08:35.808497 22992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:08:35.808560 22992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:08:35.808645 22992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:08:35.819396 22992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:36919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:36919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=39445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:08:35.820894 22992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:08:35.822376 22992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:08:35.852167 22999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:35.852749 23001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:35.852913 22998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:08:35.854142 22992 server_base.cc:1034] running on GCE node
I20260928 03:08:35.855250 22992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:08:35.857169 22992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:08:35.858456 22992 hybrid_clock.cc:648] HybridClock initialized: now 1790564915858359 us; error 53 us; skew 500 ppm
I20260928 03:08:35.858906 22992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:08:35 dist-test-slave-fjww krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564915, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:08:35.863204 22992 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:08:35.865331 22992 webserver.cc:460] Webserver started at http://127.6.192.62:39445/ using document root <none> and password file <none>
I20260928 03:08:35.866329 22992 fs_manager.cc:359] Metadata directory not provided
I20260928 03:08:35.866477 22992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:08:35.873502 22992 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:08:35.876930 23008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:35.878733 22992 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:08:35.878954 22992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "85cfa81497414912b54b17cc1fbfb2f1"
format_stamp: "Formatted at 2026-09-28 03:08:12 on dist-test-slave-fjww"
server_key: "86319ea8bf0de458565f6cd72d14daee"
server_key_iv: "38e750cf9ea6ec0c044bd9bf414369ff"
server_key_version: "encryptionkey@0"
I20260928 03:08:35.880193 22992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:08:35.895632 22992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:08:35.900441 22992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:08:35.900782 22992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:08:35.921382 22992 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:36919
I20260928 03:08:35.921406 23059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:36919 every 8 connection(s)
I20260928 03:08:35.923173 22992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:08:35.927006 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 22992
I20260928 03:08:35.928036 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:37375
--local_ip_for_outbound_sockets=127.6.192.1
--tserver_master_addrs=127.6.192.62:36919
--webserver_port=33141
--webserver_interface=127.6.192.1
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:08:35.931241 23060 sys_catalog.cc:263] Verifying existing consensus state
I20260928 03:08:35.935046 23060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1: Bootstrap starting.
I20260928 03:08:35.966113 23060 log.cc:824] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1: Log is configured to *not* fsync() on all Append() calls
I20260928 03:08:35.982833 23060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 03:08:35.983589 23060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1: Bootstrap complete.
I20260928 03:08:35.997999 23060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cfa81497414912b54b17cc1fbfb2f1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36919 } }
I20260928 03:08:35.999346 23060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85cfa81497414912b54b17cc1fbfb2f1, State: Initialized, Role: FOLLOWER
I20260928 03:08:36.000178 23060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cfa81497414912b54b17cc1fbfb2f1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36919 } }
I20260928 03:08:36.000464 23060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:08:36.000626 23060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:08:36.000811 23060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 1 FOLLOWER]: Advancing to term 2
I20260928 03:08:36.005131 23060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cfa81497414912b54b17cc1fbfb2f1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36919 } }
I20260928 03:08:36.005754 23060 leader_election.cc:302] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85cfa81497414912b54b17cc1fbfb2f1; no voters:
I20260928 03:08:36.006454 23060 leader_election.cc:288] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 03:08:36.006667 23064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 03:08:36.007908 23064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 2 LEADER]: Becoming Leader. State: Replica: 85cfa81497414912b54b17cc1fbfb2f1, State: Running, Role: LEADER
I20260928 03:08:36.008780 23060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:08:36.008634 23064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cfa81497414912b54b17cc1fbfb2f1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36919 } }
I20260928 03:08:36.011925 23066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85cfa81497414912b54b17cc1fbfb2f1. Latest consensus state: current_term: 2 leader_uuid: "85cfa81497414912b54b17cc1fbfb2f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cfa81497414912b54b17cc1fbfb2f1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36919 } } }
I20260928 03:08:36.012339 23066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [sys.catalog]: This master's current role is: LEADER
I20260928 03:08:36.012869 23065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "85cfa81497414912b54b17cc1fbfb2f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cfa81497414912b54b17cc1fbfb2f1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36919 } } }
I20260928 03:08:36.013161 23065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [sys.catalog]: This master's current role is: LEADER
I20260928 03:08:36.020764 23072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:08:36.029695 23072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:08:36.031333 23072 catalog_manager.cc:1259] Loaded cluster ID: 4d92b99f608e4b9f8258e053e76b5350
I20260928 03:08:36.031551 23072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:08:36.035943 23072 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:08:36.037756 23072 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1: Loaded TSK: 0
I20260928 03:08:36.086249 23072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1: Generated new TSK 1
I20260928 03:08:36.086740 23072 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 03:08:36.338832 23062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:08:36.339144 23062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:08:36.339293 23062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:08:36.347055 23062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:08:36.347175 23062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:08:36.347357 23062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:08:36.357996 23062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:37375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=33141
--tserver_master_addrs=127.6.192.62:36919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:08:36.359406 23062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:08:36.360867 23062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:08:36.379199 23089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:36.379207 23088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:36.380024 23091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:08:36.380601 23062 server_base.cc:1034] running on GCE node
I20260928 03:08:36.381273 23062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:08:36.382370 23062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:08:36.383623 23062 hybrid_clock.cc:648] HybridClock initialized: now 1790564916383562 us; error 49 us; skew 500 ppm
I20260928 03:08:36.383999 23062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:08:36 dist-test-slave-fjww krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564916, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:08:36.387583 23062 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:08:36.389412 23062 webserver.cc:460] Webserver started at http://127.6.192.1:33141/ using document root <none> and password file <none>
I20260928 03:08:36.390161 23062 fs_manager.cc:359] Metadata directory not provided
I20260928 03:08:36.390323 23062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:08:36.396414 23062 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:08:36.399678 23098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:36.401268 23062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 03:08:36.401465 23062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "34c53cb0f3404a6aabaa805c6ada655a"
format_stamp: "Formatted at 2026-09-28 03:08:13 on dist-test-slave-fjww"
server_key: "94755972b76708e2073e0a78b7cd4bb9"
server_key_iv: "4da2e161491ca25c55b2003915fffc43"
server_key_version: "encryptionkey@0"
I20260928 03:08:36.402550 23062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:08:36.421017 23062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:08:36.426276 23062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:08:36.426661 23062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:08:36.427965 23062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:08:36.429440 23062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:08:36.429533 23062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:36.429689 23062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:08:36.429785 23062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:36.467314 23062 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:37375
I20260928 03:08:36.467368 23210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:37375 every 8 connection(s)
I20260928 03:08:36.468928 23062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:08:36.479485 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 23062
I20260928 03:08:36.480461 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:36955
--local_ip_for_outbound_sockets=127.6.192.2
--tserver_master_addrs=127.6.192.62:36919
--webserver_port=38927
--webserver_interface=127.6.192.2
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:08:36 dist-test-slave-fjww krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564916, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:08:36.497279 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:36.472449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:49015 (local address 127.6.192.62:36919)
0928 03:08:36.473059 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:36.473072 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:36.473958 (+ 886us) server_negotiation.cc:407] Connection header received
0928 03:08:36.476356 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:36.476372 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:36.476787 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:36.477193 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:08:36.478818 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:36.480942 (+ 2124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:36.482656 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:36.483878 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:36.484099 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:36.488043 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:08:36.488058 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:08:36.488084 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:08:36.488150 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:08:36.491694 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:36.492917 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:36.492926 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:36.492933 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:36.493134 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:36.493740 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:36.493744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:36.493746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:36.494368 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:08:36.494525 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:36.495850 (+ 1325us) server_negotiation.cc:299] Negotiation successful
0928 03:08:36.496247 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":282,"threads_started":1}
I20260928 03:08:36.497546 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:36.472786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:49015)
0928 03:08:36.473724 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:36.473772 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 03:08:36.475712 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:08:36.477373 (+ 1661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:08:36.477382 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:08:36.477947 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:08:36.478532 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:36.478553 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:36.481154 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:36.481158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:36.482479 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:36.482495 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:36.484161 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:36.484168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:36.484439 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:36.485277 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:08:36.485327 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:08:36.487689 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:08:36.491909 (+ 4220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:08:36.491924 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:08:36.491945 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:08:36.492655 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:08:36.493292 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:08:36.493299 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:08:36.493306 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:08:36.493604 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:08:36.494540 (+ 936us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:08:36.494552 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:08:36.495070 (+ 518us) client_negotiation.cc:769] Sending connection context
0928 03:08:36.495433 (+ 363us) client_negotiation.cc:240] Negotiation successful
0928 03:08:36.496122 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":385,"threads_started":1}
I20260928 03:08:36.501153 23211 heartbeater.cc:347] Connected to a master server at 127.6.192.62:36919
I20260928 03:08:36.501585 23211 heartbeater.cc:464] Registering TS with master...
I20260928 03:08:36.502429 23211 heartbeater.cc:511] Master 127.6.192.62:36919 requested a full tablet report, sending...
I20260928 03:08:36.505146 23025 ts_manager.cc:194] Registered new tserver with Master: 34c53cb0f3404a6aabaa805c6ada655a (127.6.192.1:37375)
I20260928 03:08:36.506863 23025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:49015
W20260928 03:08:36.836921 23215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:08:36.837309 23215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:08:36.837522 23215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:08:36.845201 23215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:08:36.845337 23215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:08:36.845454 23215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:08:36.855940 23215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:36955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=38927
--tserver_master_addrs=127.6.192.62:36919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:08:36.857290 23215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:08:36.858671 23215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:08:36.874187 23223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:36.874104 23220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:36.874136 23221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:08:36.875480 23215 server_base.cc:1034] running on GCE node
I20260928 03:08:36.876277 23215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:08:36.877416 23215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:08:36.878661 23215 hybrid_clock.cc:648] HybridClock initialized: now 1790564916878604 us; error 44 us; skew 500 ppm
I20260928 03:08:36.879071 23215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:08:36 dist-test-slave-fjww krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564916, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:08:36.882552 23215 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:08:36.884473 23215 webserver.cc:460] Webserver started at http://127.6.192.2:38927/ using document root <none> and password file <none>
I20260928 03:08:36.885267 23215 fs_manager.cc:359] Metadata directory not provided
I20260928 03:08:36.885437 23215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:08:36.891695 23215 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:08:36.894876 23230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:36.896421 23215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:08:36.896620 23215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "4322c37b370b4da3afb2b962da7f94d3"
format_stamp: "Formatted at 2026-09-28 03:08:13 on dist-test-slave-fjww"
server_key: "e971b55baac6676975eb606409ef1376"
server_key_iv: "3c7b117086cff6806ac5ca2cfa6f9a88"
server_key_version: "encryptionkey@0"
I20260928 03:08:36.897748 23215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:08:36.919034 23215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:08:36.924463 23215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:08:36.924762 23215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:08:36.925819 23215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:08:36.927413 23215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:08:36.927596 23215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 03:08:36.927739 23215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:08:36.927829 23215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:36.965404 23215 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:36955
I20260928 03:08:36.965493 23342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:36955 every 8 connection(s)
I20260928 03:08:36.967034 23215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:08:36.970583 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 23215
I20260928 03:08:36.971302 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:43711
--local_ip_for_outbound_sockets=127.6.192.3
--tserver_master_addrs=127.6.192.62:36919
--webserver_port=41749
--webserver_interface=127.6.192.3
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 03:08:36 dist-test-slave-fjww krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564916, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:08:36.993753 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:36.970178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:41991 (local address 127.6.192.62:36919)
0928 03:08:36.970422 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:36.970429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:36.971992 (+ 1563us) server_negotiation.cc:407] Connection header received
0928 03:08:36.974201 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:36.974211 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:36.974335 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:36.974472 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:08:36.976540 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:36.977717 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:36.980626 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:36.981083 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:36.981195 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:36.986267 (+ 5072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:08:36.986275 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:08:36.986279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:08:36.986321 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:08:36.989737 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:36.990977 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:36.990982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:36.990984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:36.991131 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:36.991755 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:36.991759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:36.991761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:36.992111 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:08:36.992266 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:36.993304 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0928 03:08:36.993499 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 03:08:36.995594 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:36.970723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:41991)
0928 03:08:36.971773 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:36.971829 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 03:08:36.973838 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:08:36.974740 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:08:36.974871 (+ 131us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:08:36.975618 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:08:36.976303 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:36.976325 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:36.978037 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:36.978042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:36.980404 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:36.980428 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:36.981367 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:36.981377 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:36.981744 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:36.982579 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:08:36.982620 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:08:36.985863 (+ 3243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:08:36.989968 (+ 4105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:08:36.989982 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:08:36.990007 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:08:36.990773 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:08:36.991260 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:08:36.991267 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:08:36.991273 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:08:36.991615 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:08:36.992281 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:08:36.992294 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:08:36.992842 (+ 548us) client_negotiation.cc:769] Sending connection context
0928 03:08:36.993218 (+ 376us) client_negotiation.cc:240] Negotiation successful
0928 03:08:36.993881 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":429,"threads_started":1}
I20260928 03:08:36.997409 23343 heartbeater.cc:347] Connected to a master server at 127.6.192.62:36919
I20260928 03:08:36.997882 23343 heartbeater.cc:464] Registering TS with master...
I20260928 03:08:36.998905 23343 heartbeater.cc:511] Master 127.6.192.62:36919 requested a full tablet report, sending...
I20260928 03:08:37.000700 23025 ts_manager.cc:194] Registered new tserver with Master: 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955)
I20260928 03:08:37.001696 23025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:41991
W20260928 03:08:37.322590 23346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:08:37.322944 23346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:08:37.323123 23346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:08:37.330610 23346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:08:37.330798 23346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:08:37.330999 23346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:08:37.341171 23346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:43711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=41749
--tserver_master_addrs=127.6.192.62:36919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:08:37.342595 23346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:08:37.344056 23346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:08:37.358918 23351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:37.358911 23352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:37.358857 23354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:08:37.359750 23346 server_base.cc:1034] running on GCE node
I20260928 03:08:37.360359 23346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:08:37.361330 23346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:08:37.362538 23346 hybrid_clock.cc:648] HybridClock initialized: now 1790564917362484 us; error 44 us; skew 500 ppm
I20260928 03:08:37.362892 23346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:08:37 dist-test-slave-fjww krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564917, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:08:37.366633 23346 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:08:37.368502 23346 webserver.cc:460] Webserver started at http://127.6.192.3:41749/ using document root <none> and password file <none>
I20260928 03:08:37.369284 23346 fs_manager.cc:359] Metadata directory not provided
I20260928 03:08:37.369457 23346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:08:37.375636 23346 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:08:37.378820 23361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:37.380519 23346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:08:37.380755 23346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "7b85a9cf7cb14e0e9b734746e110f6f3"
format_stamp: "Formatted at 2026-09-28 03:08:14 on dist-test-slave-fjww"
server_key: "f47abc939473a2973891f34a30f12542"
server_key_iv: "af984c8d48215ba61c898215379ebcfe"
server_key_version: "encryptionkey@0"
I20260928 03:08:37.381902 23346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:08:37.413493 23346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:08:37.418081 23346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:08:37.418421 23346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:08:37.419587 23346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:08:37.421196 23346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:08:37.421365 23346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:37.421528 23346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:08:37.421645 23346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:37.460901 23346 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:43711
I20260928 03:08:37.460958 23473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:43711 every 8 connection(s)
I20260928 03:08:37.462551 23346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:08:37.470256 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 23346
Sep 28 03:08:37 dist-test-slave-fjww krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564917, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:08:37.494377 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:37.465593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:55967 (local address 127.6.192.62:36919)
0928 03:08:37.465786 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:37.465791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:37.467695 (+ 1904us) server_negotiation.cc:407] Connection header received
0928 03:08:37.470578 (+ 2883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:37.470585 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:37.470687 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:37.470816 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:08:37.473813 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.475531 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.479416 (+ 3885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.480232 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.480407 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:37.485126 (+ 4719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:08:37.485132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:08:37.485137 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:08:37.485182 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:08:37.489526 (+ 4344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:37.490862 (+ 1336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:37.490899 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:37.490907 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:37.491107 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:37.492008 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:37.492015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:37.492018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:37.492428 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:08:37.492877 (+ 449us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:37.493724 (+ 847us) server_negotiation.cc:299] Negotiation successful
0928 03:08:37.493996 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 03:08:37.495510 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:37.466139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:55967)
0928 03:08:37.467484 (+ 1345us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:37.467538 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 03:08:37.470188 (+ 2650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:08:37.471165 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:08:37.471180 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:08:37.472288 (+ 1108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:08:37.473449 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:37.473487 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.475919 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.475928 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:37.479034 (+ 3106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:37.479067 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.480584 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.480592 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:37.480988 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:37.481882 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:08:37.481908 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:08:37.484886 (+ 2978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:08:37.489840 (+ 4954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:08:37.489860 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:08:37.489885 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:08:37.490702 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:08:37.491284 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:08:37.491289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:08:37.491292 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:08:37.491652 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:08:37.492763 (+ 1111us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:08:37.492773 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:08:37.493251 (+ 478us) client_negotiation.cc:769] Sending connection context
0928 03:08:37.493610 (+ 359us) client_negotiation.cc:240] Negotiation successful
0928 03:08:37.494222 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"thread_start_us":654,"threads_started":1}
I20260928 03:08:37.497458 23474 heartbeater.cc:347] Connected to a master server at 127.6.192.62:36919
I20260928 03:08:37.497818 23474 heartbeater.cc:464] Registering TS with master...
I20260928 03:08:37.497828 23478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:37.471427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37068 (local address 127.6.192.62:36919)
0928 03:08:37.479928 (+ 8501us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:37.479935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:37.480377 (+ 442us) server_negotiation.cc:407] Connection header received
0928 03:08:37.480564 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:37.480570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:37.480653 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:37.480780 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:08:37.482594 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.484001 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.485745 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.486525 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.486640 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:37.490767 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:08:37.490774 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:08:37.490780 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:08:37.490841 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:08:37.493913 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:37.494940 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:37.494946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:37.494949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:37.495091 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:37.495556 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:37.495562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:37.495567 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:37.495934 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:08:37.496221 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:37.497127 (+ 906us) server_negotiation.cc:299] Negotiation successful
0928 03:08:37.497386 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8330,"thread_start_us":8038,"threads_started":1}
I20260928 03:08:37.498726 23474 heartbeater.cc:511] Master 127.6.192.62:36919 requested a full tablet report, sending...
I20260928 03:08:37.500485 23024 ts_manager.cc:194] Registered new tserver with Master: 7b85a9cf7cb14e0e9b734746e110f6f3 (127.6.192.3:43711)
I20260928 03:08:37.501533 23024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:55967
I20260928 03:08:37.510849 23211 heartbeater.cc:503] Master 127.6.192.62:36919 was elected leader, sending a full tablet report...
I20260928 03:08:37.513543 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:08:37.521545 23478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:37.514764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37072 (local address 127.6.192.62:36919)
0928 03:08:37.515046 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:37.515052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:37.515399 (+ 347us) server_negotiation.cc:407] Connection header received
0928 03:08:37.515768 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:37.515775 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:37.515953 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:37.516131 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:08:37.516643 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.517404 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.519097 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.519685 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.519834 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:37.520474 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:08:37.520700 (+ 226us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 03:08:37.521193 (+ 493us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37072: token expired
Metrics: {"server-negotiator.queue_time_us":107}
W20260928 03:08:37.521714 23478 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37072: token expired
W20260928 03:08:37.522313 22949 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.192.62:36919)
I20260928 03:08:37.536541 23478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:37.522982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37076 (local address 127.6.192.62:36919)
0928 03:08:37.523158 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:37.523163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:37.523179 (+ 16us) server_negotiation.cc:407] Connection header received
0928 03:08:37.523256 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:37.523259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:37.523319 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:37.523397 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:08:37.525037 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.525792 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.527372 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.527902 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.528006 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:37.530778 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:08:37.530786 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:08:37.530791 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:08:37.530844 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:08:37.533704 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:37.534471 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:37.534475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:37.534477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:37.534626 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:37.535068 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:37.535073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:37.535075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:37.535513 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:08:37.535729 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:37.536065 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 03:08:37.536269 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 03:08:37.540218 23025 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:08:37.547839 23025 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:08:37.582858 23487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:37.560344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49490 (local address 127.6.192.2:36955)
0928 03:08:37.569970 (+ 9626us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:37.569985 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:37.571745 (+ 1760us) server_negotiation.cc:407] Connection header received
0928 03:08:37.571892 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:37.571904 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:37.572271 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:37.572525 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:08:37.575562 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.577249 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.579420 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.581253 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.581378 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:37.581586 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:37.582386 (+ 800us) server_negotiation.cc:299] Negotiation successful
0928 03:08:37.582568 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9356,"thread_start_us":9651,"threads_started":1}
I20260928 03:08:37.582849 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:37.559991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:36955 (local address 127.0.0.1:49490)
0928 03:08:37.571047 (+ 11056us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:37.571121 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 03:08:37.571436 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:08:37.572893 (+ 1457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:08:37.572905 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:08:37.572945 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:08:37.575258 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:37.575273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.577462 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.577467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:37.579246 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:37.579259 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.581759 (+ 2500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.581766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:37.582090 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:37.582159 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 03:08:37.582283 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 03:08:37.582482 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10682,"thread_start_us":10443,"threads_started":1}
I20260928 03:08:37.584640 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:37.559991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:37375 (local address 127.0.0.1:36346)
0928 03:08:37.574756 (+ 14765us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:37.574782 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 03:08:37.574927 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:08:37.576966 (+ 2039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:08:37.576971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:08:37.577007 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:08:37.577886 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:37.577900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.579908 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.579914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:37.581910 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:37.581921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.583789 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.583796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:37.583960 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:37.583991 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 03:08:37.584098 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 03:08:37.584232 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14595,"mutex_wait_us":300,"spinlock_wait_cycles":896,"thread_start_us":14223,"threads_started":1}
I20260928 03:08:37.584710 23485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:37.560134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.6.192.1:37375)
0928 03:08:37.576207 (+ 16073us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:37.576215 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:37.576237 (+ 22us) server_negotiation.cc:407] Connection header received
0928 03:08:37.576316 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:37.576323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:37.576530 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:37.576688 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:08:37.578078 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.579742 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.582070 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.583569 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.583706 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:37.584038 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:37.584173 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 03:08:37.584355 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15900,"thread_start_us":15750,"threads_started":1}
I20260928 03:08:37.587715 23278 tablet_service.cc:1467] Processing CreateTablet for tablet b871e2cdc61d4245bf951e065020a1d9 (DEFAULT_TABLE table=test-table [id=78096a3ca30d4e018d77f4f9029b75e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:08:37.588511 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:37.561959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:43711 (local address 127.0.0.1:54150)
0928 03:08:37.575750 (+ 13791us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:37.575771 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:08:37.575920 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:08:37.576966 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:08:37.576971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:08:37.577014 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:08:37.577778 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:37.577798 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.579878 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.579884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:37.582170 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:37.582182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.587645 (+ 5463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.587655 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:37.587797 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:37.587824 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 03:08:37.587956 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 03:08:37.588100 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13653,"mutex_wait_us":46,"spinlock_wait_cycles":7808,"thread_start_us":13521,"threads_started":1}
I20260928 03:08:37.588580 23489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:37.562844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.6.192.3:43711)
0928 03:08:37.575224 (+ 12380us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:37.575238 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:37.576060 (+ 822us) server_negotiation.cc:407] Connection header received
0928 03:08:37.576172 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:37.576182 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:37.576451 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:37.576655 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:08:37.578036 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.579733 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.582336 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.583845 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.583974 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:37.584209 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:37.588012 (+ 3803us) server_negotiation.cc:299] Negotiation successful
0928 03:08:37.588196 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12170,"thread_start_us":11982,"threads_started":1}
I20260928 03:08:37.588622 23146 tablet_service.cc:1467] Processing CreateTablet for tablet b871e2cdc61d4245bf951e065020a1d9 (DEFAULT_TABLE table=test-table [id=78096a3ca30d4e018d77f4f9029b75e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:08:37.589571 23278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b871e2cdc61d4245bf951e065020a1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:08:37.590189 23146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b871e2cdc61d4245bf951e065020a1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:08:37.591871 23409 tablet_service.cc:1467] Processing CreateTablet for tablet b871e2cdc61d4245bf951e065020a1d9 (DEFAULT_TABLE table=test-table [id=78096a3ca30d4e018d77f4f9029b75e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:08:37.593181 23409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b871e2cdc61d4245bf951e065020a1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:08:37.607475 23490 tablet_bootstrap.cc:492] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3: Bootstrap starting.
I20260928 03:08:37.610571 23490 tablet_bootstrap.cc:654] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3: Neither blocks nor log segments found. Creating new log.
I20260928 03:08:37.611827 23490 log.cc:824] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3: Log is configured to *not* fsync() on all Append() calls
I20260928 03:08:37.612066 23491 tablet_bootstrap.cc:492] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3: Bootstrap starting.
I20260928 03:08:37.612232 23492 tablet_bootstrap.cc:492] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a: Bootstrap starting.
I20260928 03:08:37.616987 23491 tablet_bootstrap.cc:654] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3: Neither blocks nor log segments found. Creating new log.
I20260928 03:08:37.617144 23492 tablet_bootstrap.cc:654] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a: Neither blocks nor log segments found. Creating new log.
I20260928 03:08:37.618155 23491 log.cc:824] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3: Log is configured to *not* fsync() on all Append() calls
I20260928 03:08:37.618741 23492 log.cc:824] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a: Log is configured to *not* fsync() on all Append() calls
I20260928 03:08:37.621275 23490 tablet_bootstrap.cc:492] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3: No bootstrap required, opened a new log
I20260928 03:08:37.621500 23491 tablet_bootstrap.cc:492] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3: No bootstrap required, opened a new log
I20260928 03:08:37.621606 23490 ts_tablet_manager.cc:1397] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.005s
I20260928 03:08:37.621791 23491 ts_tablet_manager.cc:1397] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260928 03:08:37.623811 23492 tablet_bootstrap.cc:492] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a: No bootstrap required, opened a new log
I20260928 03:08:37.624249 23492 ts_tablet_manager.cc:1397] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260928 03:08:37.630007 23491 raft_consensus.cc:348] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } }
I20260928 03:08:37.630436 23491 raft_consensus.cc:374] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:08:37.630599 23491 raft_consensus.cc:729] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4322c37b370b4da3afb2b962da7f94d3, State: Initialized, Role: FOLLOWER
I20260928 03:08:37.631256 23491 consensus_queue.cc:260] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } }
I20260928 03:08:37.632483 23343 heartbeater.cc:503] Master 127.6.192.62:36919 was elected leader, sending a full tablet report...
I20260928 03:08:37.632712 23490 raft_consensus.cc:348] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } }
I20260928 03:08:37.633188 23490 raft_consensus.cc:374] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:08:37.633306 23490 raft_consensus.cc:729] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b85a9cf7cb14e0e9b734746e110f6f3, State: Initialized, Role: FOLLOWER
I20260928 03:08:37.633944 23490 consensus_queue.cc:260] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } }
I20260928 03:08:37.634830 23491 ts_tablet_manager.cc:1428] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3: Time spent starting tablet: real 0.013s user 0.007s sys 0.003s
I20260928 03:08:37.635579 23492 raft_consensus.cc:348] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } }
I20260928 03:08:37.636124 23492 raft_consensus.cc:374] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:08:37.636202 23474 heartbeater.cc:503] Master 127.6.192.62:36919 was elected leader, sending a full tablet report...
I20260928 03:08:37.636264 23492 raft_consensus.cc:729] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34c53cb0f3404a6aabaa805c6ada655a, State: Initialized, Role: FOLLOWER
I20260928 03:08:37.636936 23490 ts_tablet_manager.cc:1428] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260928 03:08:37.637010 23492 consensus_queue.cc:260] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } }
I20260928 03:08:37.639693 23492 ts_tablet_manager.cc:1428] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260928 03:08:37.714484 23475 tablet.cc:2369] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:08:37.719197 23344 tablet.cc:2369] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:08:37.721637 23212 tablet.cc:2369] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:08:37.740620 23497 raft_consensus.cc:482] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:08:37.740984 23497 raft_consensus.cc:504] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } }
I20260928 03:08:37.742738 23497 leader_election.cc:288] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34c53cb0f3404a6aabaa805c6ada655a (127.6.192.1:37375), 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955)
I20260928 03:08:37.750010 23487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:37.743345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:44687 (local address 127.6.192.2:36955)
0928 03:08:37.743558 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:37.743567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:37.743728 (+ 161us) server_negotiation.cc:407] Connection header received
0928 03:08:37.743952 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:37.743957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:37.744038 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:37.744159 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:08:37.744753 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.745759 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.748086 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.748969 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.749155 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:37.749274 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:37.749564 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 03:08:37.749705 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 03:08:37.750059 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:37.743177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:36955 (local address 127.6.192.3:44687)
0928 03:08:37.743578 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:37.743609 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 03:08:37.743793 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:08:37.744211 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:08:37.744214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:08:37.744237 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:08:37.744587 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:37.744601 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.745917 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.745924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:37.747884 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:37.747899 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.749161 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.749168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:37.749384 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:37.749410 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 03:08:37.749530 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 03:08:37.749684 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":3,"spinlock_wait_cycles":25472}
I20260928 03:08:37.750726 23485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:37.743237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:54141 (local address 127.6.192.1:37375)
0928 03:08:37.743504 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:37.743513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:37.744319 (+ 806us) server_negotiation.cc:407] Connection header received
0928 03:08:37.744469 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:37.744475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:37.744557 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:37.744665 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:08:37.745458 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.746769 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.748882 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.749686 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.749856 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:37.749966 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:37.750313 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 03:08:37.750450 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 03:08:37.751492 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:37.743207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:37375 (local address 127.6.192.3:54141)
0928 03:08:37.744183 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:37.744200 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:08:37.744320 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:08:37.744833 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:08:37.744841 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:08:37.744872 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:08:37.745238 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:37.745251 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.746947 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.746951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:37.748696 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:08:37.748709 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.749911 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.749917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:08:37.750083 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:37.750140 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 03:08:37.750943 (+ 803us) client_negotiation.cc:240] Negotiation successful
0928 03:08:37.751086 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":3,"thread_start_us":494,"threads_started":1}
I20260928 03:08:37.751272 23298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b871e2cdc61d4245bf951e065020a1d9" candidate_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4322c37b370b4da3afb2b962da7f94d3" is_pre_election: true
I20260928 03:08:37.751890 23298 raft_consensus.cc:2454] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b85a9cf7cb14e0e9b734746e110f6f3 in term 0.
I20260928 03:08:37.752684 23166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b871e2cdc61d4245bf951e065020a1d9" candidate_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34c53cb0f3404a6aabaa805c6ada655a" is_pre_election: true
I20260928 03:08:37.752689 23365 leader_election.cc:302] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4322c37b370b4da3afb2b962da7f94d3, 7b85a9cf7cb14e0e9b734746e110f6f3; no voters:
I20260928 03:08:37.753207 23166 raft_consensus.cc:2454] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b85a9cf7cb14e0e9b734746e110f6f3 in term 0.
I20260928 03:08:37.753371 23497 raft_consensus.cc:2781] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:08:37.753559 23497 raft_consensus.cc:482] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:08:37.753712 23497 raft_consensus.cc:3037] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:08:37.756309 23497 raft_consensus.cc:504] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } }
I20260928 03:08:37.757059 23497 leader_election.cc:288] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [CANDIDATE]: Term 1 election: Requested vote from peers 34c53cb0f3404a6aabaa805c6ada655a (127.6.192.1:37375), 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955)
I20260928 03:08:37.757556 23166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b871e2cdc61d4245bf951e065020a1d9" candidate_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34c53cb0f3404a6aabaa805c6ada655a"
I20260928 03:08:37.757613 23298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b871e2cdc61d4245bf951e065020a1d9" candidate_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4322c37b370b4da3afb2b962da7f94d3"
I20260928 03:08:37.757788 23298 raft_consensus.cc:3037] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:08:37.757803 23166 raft_consensus.cc:3037] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:08:37.760337 23298 raft_consensus.cc:2454] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b85a9cf7cb14e0e9b734746e110f6f3 in term 1.
I20260928 03:08:37.760342 23166 raft_consensus.cc:2454] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b85a9cf7cb14e0e9b734746e110f6f3 in term 1.
I20260928 03:08:37.760844 23365 leader_election.cc:302] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4322c37b370b4da3afb2b962da7f94d3, 7b85a9cf7cb14e0e9b734746e110f6f3; no voters:
I20260928 03:08:37.761204 23497 raft_consensus.cc:2781] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:08:37.761839 23497 raft_consensus.cc:686] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 1 LEADER]: Becoming Leader. State: Replica: 7b85a9cf7cb14e0e9b734746e110f6f3, State: Running, Role: LEADER
I20260928 03:08:37.762494 23497 consensus_queue.cc:237] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } }
I20260928 03:08:37.767961 23024 catalog_manager.cc:5495] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b85a9cf7cb14e0e9b734746e110f6f3 (127.6.192.3). New cstate: current_term: 1 leader_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } health_report { overall_health: HEALTHY } } }
I20260928 03:08:37.828094 23489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:37.822245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54158 (local address 127.6.192.3:43711)
0928 03:08:37.822609 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:37.822620 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:37.822644 (+ 24us) server_negotiation.cc:407] Connection header received
0928 03:08:37.822911 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:37.822915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:37.823010 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:37.823179 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:08:37.823635 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.824353 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.825753 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.826181 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.826325 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:37.826906 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:08:37.827473 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:08:37.827644 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:37.827724 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 03:08:37.827813 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260928 03:08:37.836993 23166 raft_consensus.cc:1261] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Refusing update from remote peer 7b85a9cf7cb14e0e9b734746e110f6f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:08:37.836993 23298 raft_consensus.cc:1261] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3 [term 1 FOLLOWER]: Refusing update from remote peer 7b85a9cf7cb14e0e9b734746e110f6f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:08:37.838335 23500 consensus_queue.cc:1035] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:08:37.838701 23497 consensus_queue.cc:1035] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:08:37.852456 23502 mvcc.cc:205] Tried to move back new op lower bound from 7334153903449276416 to 7334153903160025088. Current Snapshot: MvccSnapshot[applied={T|T < 7334153903449276416}]
I20260928 03:08:37.869012 23025 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:08:37.869349 23025 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:08:37.873543 23025 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1: Sending DeleteTablet for 3 replicas of tablet b871e2cdc61d4245bf951e065020a1d9
I20260928 03:08:37.874861 23146 tablet_service.cc:1514] Processing DeleteTablet for tablet b871e2cdc61d4245bf951e065020a1d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:08:37 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:36346
I20260928 03:08:37.874871 23409 tablet_service.cc:1514] Processing DeleteTablet for tablet b871e2cdc61d4245bf951e065020a1d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:08:37 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:54150
I20260928 03:08:37.875726 23278 tablet_service.cc:1514] Processing DeleteTablet for tablet b871e2cdc61d4245bf951e065020a1d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:08:37 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:49490
I20260928 03:08:37.877012 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 23062
I20260928 03:08:37.885459 23514 tablet_replica.cc:330] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3: stopping tablet replica
I20260928 03:08:37.887356 23514 raft_consensus.cc:2229] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:08:37.888756 23514 raft_consensus.cc:2258] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:08:37.893373 23513 tablet_replica.cc:330] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3: stopping tablet replica
I20260928 03:08:37.893724 23514 ts_tablet_manager.cc:1905] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:08:37.893853 23513 raft_consensus.cc:2229] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:08:37.894596 23513 raft_consensus.cc:2258] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:08:37.897161 23512 tablet_replica.cc:330] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a: stopping tablet replica
I20260928 03:08:37.897627 23512 raft_consensus.cc:2229] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:08:37.898185 23512 raft_consensus.cc:2258] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:08:37.900466 23513 ts_tablet_manager.cc:1905] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:08:37.901157 23514 ts_tablet_manager.cc:1918] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:08:37.901489 23514 log.cc:1196] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/b871e2cdc61d4245bf951e065020a1d9
I20260928 03:08:37.901686 23485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:37.878536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36360 (local address 127.6.192.1:37375)
0928 03:08:37.878716 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:37.878722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:37.878740 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:08:37.878813 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:37.878818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:37.878924 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:37.879026 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:08:37.881104 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.882326 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.884497 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:37.885186 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:37.885319 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:37.889862 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:08:37.889884 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:08:37.889905 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:08:37.889962 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:08:37.896660 (+ 6698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:37.897908 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:37.897919 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:37.897924 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:37.898136 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:37.898978 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:37.898983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:37.898987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:37.899488 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:08:37.899803 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:37.901061 (+ 1258us) server_negotiation.cc:299] Negotiation successful
0928 03:08:37.901303 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 03:08:37.902211 23514 ts_tablet_manager.cc:1939] T b871e2cdc61d4245bf951e065020a1d9 P 4322c37b370b4da3afb2b962da7f94d3: Deleting consensus metadata
I20260928 03:08:37.902735 23206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:08:37.908488 23012 catalog_manager.cc:4841] TS 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955): tablet b871e2cdc61d4245bf951e065020a1d9 (table test-table [id=78096a3ca30d4e018d77f4f9029b75e7]) successfully deleted
I20260928 03:08:37.909399 23513 ts_tablet_manager.cc:1918] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:08:37.909611 23513 log.cc:1196] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/b871e2cdc61d4245bf951e065020a1d9
I20260928 03:08:37.910127 23513 ts_tablet_manager.cc:1939] T b871e2cdc61d4245bf951e065020a1d9 P 7b85a9cf7cb14e0e9b734746e110f6f3: Deleting consensus metadata
I20260928 03:08:37.911722 23010 catalog_manager.cc:4841] TS 7b85a9cf7cb14e0e9b734746e110f6f3 (127.6.192.3:43711): tablet b871e2cdc61d4245bf951e065020a1d9 (table test-table [id=78096a3ca30d4e018d77f4f9029b75e7]) successfully deleted
I20260928 03:08:38.020362 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 23062
I20260928 03:08:38.038167 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 23215
W20260928 03:08:38.038342 23009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.192.1:37375 (error 108)
I20260928 03:08:38.040417 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:38.039902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:37375 (local address 127.0.0.1:36366)
0928 03:08:38.040100 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:38.040227 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:37375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 03:08:38.040704 23009 catalog_manager.cc:4584] TS 34c53cb0f3404a6aabaa805c6ada655a (127.6.192.1:37375): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b871e2cdc61d4245bf951e065020a1d9: Network error: Client connection negotiation failed: client connection to 127.6.192.1:37375: connect: Connection refused (error 111)
I20260928 03:08:38.055709 23487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:38.039617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49502 (local address 127.6.192.2:36955)
0928 03:08:38.039791 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:38.039797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:38.039816 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:08:38.039893 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:38.039898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:38.039991 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:38.040096 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:08:38.041825 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:38.043146 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:38.044609 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:38.045325 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:38.045477 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:38.048207 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:08:38.048227 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:08:38.048249 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:08:38.048297 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:08:38.052583 (+ 4286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:38.053347 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:38.053358 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:38.053362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:38.053526 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:38.053905 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:38.053911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:38.053915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:38.054437 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:08:38.054778 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:38.054997 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 03:08:38.055252 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 03:08:38.056533 23338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:08:38.094756 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:38.094200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:37375 (local address 127.0.0.1:36374)
0928 03:08:38.094459 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:38.094583 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:37375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 03:08:38.096045 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:38.095585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:37375 (local address 127.0.0.1:36386)
0928 03:08:38.095767 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:38.095879 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:37375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 03:08:38.150339 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 23215
I20260928 03:08:38.156736 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:38.156114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:37375 (local address 127.0.0.1:36388)
0928 03:08:38.156385 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:38.156503 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:37375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 03:08:38.158021 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:38.157559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:37375 (local address 127.0.0.1:36404)
0928 03:08:38.157766 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:38.157870 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:37375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 03:08:38.168737 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 23346
I20260928 03:08:38.186687 23489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:38.169751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54166 (local address 127.6.192.3:43711)
0928 03:08:38.169946 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 03:08:38.169951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:08:38.169969 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:08:38.170177 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:08:38.170184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:08:38.170247 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:08:38.170328 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:08:38.172554 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:38.173367 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:38.175641 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:08:38.176087 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:08:38.176191 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:08:38.179861 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:08:38.179872 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:08:38.179888 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:08:38.179930 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:08:38.183050 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:38.184070 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:38.184077 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:38.184079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:38.184239 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:08:38.184770 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:08:38.184777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:08:38.184780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:08:38.185304 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:08:38.185691 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0928 03:08:38.186040 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 03:08:38.186244 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 03:08:38.188012 23469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:08:38.238978 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:38.238417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:37375 (local address 127.0.0.1:36416)
0928 03:08:38.238659 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:38.238761 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:37375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 03:08:38.240471 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:38.239974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:37375 (local address 127.0.0.1:36420)
0928 03:08:38.240256 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:38.240338 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:37375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260928 03:08:38.309782 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 23346
I20260928 03:08:38.329730 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 22992
I20260928 03:08:38.331177 23055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:08:38.446599 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:38.445881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:37375 (local address 127.0.0.1:36432)
0928 03:08:38.446101 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:38.446298 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:37375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 03:08:38.446930 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 22992
I20260928 03:08:58.469292 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:36919
--webserver_interface=127.6.192.62
--webserver_port=39445
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:36919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:08:58.842274 23520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:08:58.842711 23520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:08:58.842945 23520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:08:58.850729 23520 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 03:08:58.850917 23520 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 03:08:58.851029 23520 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 03:08:58.851125 23520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:08:58.851235 23520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:08:58.851372 23520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:08:58.851506 23520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:08:58.851604 23520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:08:58.862454 23520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:36919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:36919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=39445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:08:58.864058 23520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:08:58.865643 23520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:08:58.879962 23525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:58.879968 23526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:58.880024 23528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:08:58.880466 23520 server_base.cc:1034] running on GCE node
I20260928 03:08:58.881639 23520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:08:58.883231 23520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:08:58.884490 23520 hybrid_clock.cc:648] HybridClock initialized: now 1790564938884436 us; error 73 us; skew 500 ppm
I20260928 03:08:58.884891 23520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:08:58 dist-test-slave-fjww krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564938, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:08:58.893579 23520 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:08:58.895524 23520 webserver.cc:460] Webserver started at http://127.6.192.62:39445/ using document root <none> and password file <none>
I20260928 03:08:58.896332 23520 fs_manager.cc:359] Metadata directory not provided
I20260928 03:08:58.896431 23520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:08:58.902964 23520 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 03:08:58.906489 23535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:58.908591 23520 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:08:58.908836 23520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "85cfa81497414912b54b17cc1fbfb2f1"
format_stamp: "Formatted at 2026-09-28 03:08:12 on dist-test-slave-fjww"
server_key: "86319ea8bf0de458565f6cd72d14daee"
server_key_iv: "38e750cf9ea6ec0c044bd9bf414369ff"
server_key_version: "encryptionkey@0"
I20260928 03:08:58.910058 23520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:08:58.933336 23520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:08:58.938841 23520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:08:58.939188 23520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:08:58.960601 23520 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:36919
I20260928 03:08:58.960660 23586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:36919 every 8 connection(s)
I20260928 03:08:58.962253 23520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:08:58.967598 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 23520
I20260928 03:08:58.968667 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:37375
--local_ip_for_outbound_sockets=127.6.192.1
--tserver_master_addrs=127.6.192.62:36919
--webserver_port=33141
--webserver_interface=127.6.192.1
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:08:58.971107 23587 sys_catalog.cc:263] Verifying existing consensus state
I20260928 03:08:58.976894 23587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1: Bootstrap starting.
I20260928 03:08:58.999050 23587 log.cc:824] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1: Log is configured to *not* fsync() on all Append() calls
I20260928 03:08:59.017369 23587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 03:08:59.018015 23587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1: Bootstrap complete.
I20260928 03:08:59.027810 23587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cfa81497414912b54b17cc1fbfb2f1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36919 } }
I20260928 03:08:59.028954 23587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85cfa81497414912b54b17cc1fbfb2f1, State: Initialized, Role: FOLLOWER
I20260928 03:08:59.029726 23587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cfa81497414912b54b17cc1fbfb2f1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36919 } }
I20260928 03:08:59.029986 23587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:08:59.030117 23587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:08:59.030356 23587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 2 FOLLOWER]: Advancing to term 3
I20260928 03:08:59.034572 23587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cfa81497414912b54b17cc1fbfb2f1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36919 } }
I20260928 03:08:59.035151 23587 leader_election.cc:302] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85cfa81497414912b54b17cc1fbfb2f1; no voters:
I20260928 03:08:59.035938 23587 leader_election.cc:288] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 03:08:59.036075 23591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 03:08:59.037314 23591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [term 3 LEADER]: Becoming Leader. State: Replica: 85cfa81497414912b54b17cc1fbfb2f1, State: Running, Role: LEADER
I20260928 03:08:59.037904 23591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cfa81497414912b54b17cc1fbfb2f1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36919 } }
I20260928 03:08:59.038308 23587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:08:59.040918 23592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "85cfa81497414912b54b17cc1fbfb2f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cfa81497414912b54b17cc1fbfb2f1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36919 } } }
I20260928 03:08:59.041396 23592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [sys.catalog]: This master's current role is: LEADER
I20260928 03:08:59.043131 23593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85cfa81497414912b54b17cc1fbfb2f1. Latest consensus state: current_term: 3 leader_uuid: "85cfa81497414912b54b17cc1fbfb2f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cfa81497414912b54b17cc1fbfb2f1" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36919 } } }
I20260928 03:08:59.043797 23593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1 [sys.catalog]: This master's current role is: LEADER
I20260928 03:08:59.049490 23597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:08:59.060679 23597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:08:59.062716 23597 catalog_manager.cc:1259] Loaded cluster ID: 4d92b99f608e4b9f8258e053e76b5350
I20260928 03:08:59.062862 23597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:08:59.069032 23597 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:08:59.071600 23597 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1: Loaded TSK: 0
I20260928 03:08:59.071821 23597 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85cfa81497414912b54b17cc1fbfb2f1: Loaded TSK: 1
F20260928 03:08:59.075524 23597 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff4d153d9ad google::LogMessage::Fail() at ??:0
@ 0x7ff4d154088a google::LogMessage::SendToLog() at ??:0
@ 0x7ff4d153d4de google::LogMessage::Flush() at ??:0
@ 0x7ff4d153ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff4d52a17d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff4d529073e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff4e6f6c1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff4e6f6b783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff4e6ff72a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7ff4e6ff70c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7ff4e70415e9 std::function<>::operator()() at ??:0
@ 0x7ff4e6f6fcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff4e6f6dc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff4e6ff5610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff4e6ff5439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff4d4189345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff4d418c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff4d418c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff4d4167eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff4d74416db start_thread at ??:0
@ 0x7ff4d03d761f clone at ??:0
W20260928 03:08:59.407497 23589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:08:59.407899 23589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:08:59.408075 23589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:08:59.416100 23589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:08:59.416260 23589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:08:59.416440 23589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:08:59.426988 23589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:37375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=33141
--tserver_master_addrs=127.6.192.62:36919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:08:59.428516 23589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:08:59.429986 23589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:08:59.444705 23637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:59.444869 23634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:59.444872 23635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:08:59.446830 23589 server_base.cc:1034] running on GCE node
I20260928 03:08:59.447646 23589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:08:59.448763 23589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:08:59.450027 23589 hybrid_clock.cc:648] HybridClock initialized: now 1790564939449954 us; error 65 us; skew 500 ppm
I20260928 03:08:59.450385 23589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:08:59 dist-test-slave-fjww krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564939, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:08:59.454005 23589 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:08:59.455811 23589 webserver.cc:460] Webserver started at http://127.6.192.1:33141/ using document root <none> and password file <none>
I20260928 03:08:59.456568 23589 fs_manager.cc:359] Metadata directory not provided
I20260928 03:08:59.456753 23589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:08:59.462805 23589 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 03:08:59.466557 23644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:59.468403 23589 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260928 03:08:59.468633 23589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "34c53cb0f3404a6aabaa805c6ada655a"
format_stamp: "Formatted at 2026-09-28 03:08:13 on dist-test-slave-fjww"
server_key: "94755972b76708e2073e0a78b7cd4bb9"
server_key_iv: "4da2e161491ca25c55b2003915fffc43"
server_key_version: "encryptionkey@0"
I20260928 03:08:59.469812 23589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:08:59.495050 23589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:08:59.500420 23589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:08:59.500751 23589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:08:59.501880 23589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:08:59.504022 23651 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260928 03:08:59.510824 23589 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 03:08:59.511029 23589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.000s
I20260928 03:08:59.511209 23589 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 03:08:59.514922 23589 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 03:08:59.515064 23589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.001s
I20260928 03:08:59.515549 23651 tablet_bootstrap.cc:492] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a: Bootstrap starting.
I20260928 03:08:59.525194 23651 log.cc:824] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a: Log is configured to *not* fsync() on all Append() calls
I20260928 03:08:59.543381 23651 tablet_bootstrap.cc:492] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 03:08:59.544658 23651 tablet_bootstrap.cc:492] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a: Bootstrap complete.
I20260928 03:08:59.545291 23651 ts_tablet_manager.cc:1397] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a: Time spent bootstrapping tablet: real 0.030s user 0.020s sys 0.008s
I20260928 03:08:59.555022 23651 raft_consensus.cc:348] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } }
I20260928 03:08:59.556838 23651 raft_consensus.cc:729] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34c53cb0f3404a6aabaa805c6ada655a, State: Initialized, Role: FOLLOWER
I20260928 03:08:59.557704 23651 consensus_queue.cc:260] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } }
I20260928 03:08:59.559393 23651 ts_tablet_manager.cc:1428] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260928 03:08:59.566187 23589 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:37375
I20260928 03:08:59.566226 23759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:37375 every 8 connection(s)
I20260928 03:08:59.567724 23589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:08:59.571048 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 23589
I20260928 03:08:59.571857 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:36955
--local_ip_for_outbound_sockets=127.6.192.2
--tserver_master_addrs=127.6.192.62:36919
--webserver_port=38927
--webserver_interface=127.6.192.2
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:08:59.573108 23762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:59.570790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:59307)
0928 03:08:59.571792 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:59.572081 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":426,"threads_started":1}
W20260928 03:08:59.574064 23760 heartbeater.cc:644] Failed to heartbeat to 127.6.192.62:36919 (0 consecutive failures): Network error: Failed to ping master at 127.6.192.62:36919: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
I20260928 03:08:59.575179 23762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:59.574735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:47559)
0928 03:08:59.574987 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:59.575065 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260928 03:08:59.576196 23762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:08:59.575817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:38885)
0928 03:08:59.576001 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 03:08:59.576074 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 03:08:59.576444 23760 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 03:08:59.924635 23763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:08:59.924962 23763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:08:59.925130 23763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:08:59.933185 23763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:08:59.933321 23763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:08:59.933511 23763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:08:59.943989 23763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:36955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=38927
--tserver_master_addrs=127.6.192.62:36919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:08:59.945392 23763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:08:59.946805 23763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:08:59.961550 23769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:59.961565 23771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:08:59.962246 23768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:08:59.962507 23763 server_base.cc:1034] running on GCE node
I20260928 03:08:59.963191 23763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:08:59.964602 23763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:08:59.965862 23763 hybrid_clock.cc:648] HybridClock initialized: now 1790564939965861 us; error 114 us; skew 500 ppm
I20260928 03:08:59.966292 23763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:08:59 dist-test-slave-fjww krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564939, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:08:59.969774 23763 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:08:59.971405 23763 webserver.cc:460] Webserver started at http://127.6.192.2:38927/ using document root <none> and password file <none>
I20260928 03:08:59.972254 23763 fs_manager.cc:359] Metadata directory not provided
I20260928 03:08:59.972424 23763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:08:59.978744 23763 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 03:08:59.982072 23778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:08:59.983630 23763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:08:59.983829 23763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "4322c37b370b4da3afb2b962da7f94d3"
format_stamp: "Formatted at 2026-09-28 03:08:13 on dist-test-slave-fjww"
server_key: "e971b55baac6676975eb606409ef1376"
server_key_iv: "3c7b117086cff6806ac5ca2cfa6f9a88"
server_key_version: "encryptionkey@0"
I20260928 03:08:59.984947 23763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:08:59.997015 23763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:00.003664 23763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:00.004021 23763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:00.005227 23763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:00.006903 23763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:00.007055 23763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:00.007206 23763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:00.007303 23763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:00.046583 23763 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:36955
I20260928 03:09:00.046643 23890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:36955 every 8 connection(s)
I20260928 03:09:00.048275 23763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:09:00.049779 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 23763
I20260928 03:09:00.050525 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:43711
--local_ip_for_outbound_sockets=127.6.192.3
--tserver_master_addrs=127.6.192.62:36919
--webserver_port=41749
--webserver_interface=127.6.192.3
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:09:00.054126 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:00.052008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:60039)
0928 03:09:00.052969 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:00.053203 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":390,"threads_started":1}
W20260928 03:09:00.055043 23891 heartbeater.cc:644] Failed to heartbeat to 127.6.192.62:36919 (0 consecutive failures): Network error: Failed to ping master at 127.6.192.62:36919: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
I20260928 03:09:00.056066 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:00.055695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:46175)
0928 03:09:00.055860 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:00.055948 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 03:09:00.057164 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:00.056615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:39063)
0928 03:09:00.056824 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:00.056947 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 03:09:00.057495 23891 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 03:09:00.393659 23894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:00.394032 23894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:00.394212 23894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:00.401610 23894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:00.401782 23894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:00.401988 23894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:09:00.411896 23894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:43711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=41749
--tserver_master_addrs=127.6.192.62:36919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:00.413280 23894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:00.414672 23894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:00.428926 23899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:00.429021 23902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:00.428884 23900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:00.429724 23894 server_base.cc:1034] running on GCE node
I20260928 03:09:00.430389 23894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:00.431408 23894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:00.432713 23894 hybrid_clock.cc:648] HybridClock initialized: now 1790564940432662 us; error 50 us; skew 500 ppm
I20260928 03:09:00.433015 23894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:00 dist-test-slave-fjww krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564940, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:00.436374 23894 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:09:00.438081 23894 webserver.cc:460] Webserver started at http://127.6.192.3:41749/ using document root <none> and password file <none>
I20260928 03:09:00.438800 23894 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:00.438964 23894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:00.444986 23894 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 03:09:00.448230 23909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:00.449712 23894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 03:09:00.449905 23894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "7b85a9cf7cb14e0e9b734746e110f6f3"
format_stamp: "Formatted at 2026-09-28 03:08:14 on dist-test-slave-fjww"
server_key: "f47abc939473a2973891f34a30f12542"
server_key_iv: "af984c8d48215ba61c898215379ebcfe"
server_key_version: "encryptionkey@0"
I20260928 03:09:00.450976 23894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:00.479214 23894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:00.484459 23894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:00.484786 23894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:00.485879 23894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:00.487376 23894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:00.487534 23894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 03:09:00.487663 23894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:00.487754 23894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:00.524636 23894 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:43711
I20260928 03:09:00.524835 24021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:43711 every 8 connection(s)
I20260928 03:09:00.526141 23894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:09:00.529264 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 23894
I20260928 03:09:00.532058 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:00.529230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:50739)
0928 03:09:00.530254 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:00.530561 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":364,"threads_started":1}
W20260928 03:09:00.532822 24022 heartbeater.cc:644] Failed to heartbeat to 127.6.192.62:36919 (0 consecutive failures): Network error: Failed to ping master at 127.6.192.62:36919: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
I20260928 03:09:00.533840 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:00.533384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:38255)
0928 03:09:00.533577 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:00.533650 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 03:09:00.534962 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:00.534520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:39497)
0928 03:09:00.534698 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:00.534810 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260928 03:09:00.535310 24022 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 03:09:00.578081 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:00.577120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:37201)
0928 03:09:00.577755 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:00.577835 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":351,"threads_started":1}
I20260928 03:09:01.059238 24027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:01.058301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:37103)
0928 03:09:01.058867 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:01.058977 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":288,"threads_started":1}
I20260928 03:09:01.345199 24028 raft_consensus.cc:482] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:09:01.345544 24028 raft_consensus.cc:504] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } }
I20260928 03:09:01.347277 24028 leader_election.cc:288] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955), 7b85a9cf7cb14e0e9b734746e110f6f3 (127.6.192.3:43711)
Sep 28 03:09:01 dist-test-slave-fjww krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564939, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.3@KRBTEST.COM
Sep 28 03:09:01 dist-test-slave-fjww krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564939, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.2@KRBTEST.COM
I20260928 03:09:01.374315 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:01.347829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:43711 (local address 127.6.192.1:49587)
0928 03:09:01.350039 (+ 2210us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:01.350069 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:01.350192 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:01.354476 (+ 4284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:01.354481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:01.355489 (+ 1008us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:01.356691 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:01.356737 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:01.358737 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:01.358742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:01.360345 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:01.360360 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:01.361432 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:01.361439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:01.361754 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:01.362576 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:09:01.362601 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:01.365606 (+ 3005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:01.369844 (+ 4238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:01.369858 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:01.369878 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:01.370558 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:01.371154 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:01.371159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:01.371162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:01.371536 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:01.372545 (+ 1009us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:01.372550 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:01.372924 (+ 374us) client_negotiation.cc:769] Sending connection context
0928 03:09:01.373192 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 03:09:01.373788 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2068,"mutex_wait_us":228,"spinlock_wait_cycles":35968,"thread_start_us":1912,"threads_started":1}
I20260928 03:09:01.374435 24029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:01.347535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:36955 (local address 127.6.192.1:34019)
0928 03:09:01.348318 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:01.348366 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:01.349113 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:01.354145 (+ 5032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:01.354155 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:01.355489 (+ 1334us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:01.357092 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:01.357103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:01.358735 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:01.358739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:01.360364 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:01.360378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:01.361426 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:01.361453 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:01.361755 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:01.362635 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:09:01.362652 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:01.365814 (+ 3162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:01.370293 (+ 4479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:01.370298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:01.370300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:01.370780 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:01.371397 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:01.371401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:01.371403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:01.371668 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:01.372528 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:01.372537 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:01.372917 (+ 380us) client_negotiation.cc:769] Sending connection context
0928 03:09:01.373192 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 03:09:01.373953 (+ 761us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":593,"spinlock_wait_cycles":16000,"thread_start_us":745,"threads_started":1}
I20260928 03:09:01.374485 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:01.348097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:49587 (local address 127.6.192.3:43711)
0928 03:09:01.350669 (+ 2572us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:01.350681 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:01.350712 (+ 31us) server_negotiation.cc:407] Connection header received
0928 03:09:01.352928 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:01.352943 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:01.353519 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:01.353949 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:01.356996 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:01.358520 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:01.360500 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:01.361162 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:01.361377 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:01.365983 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:01.365995 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:01.366015 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:01.366079 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:01.369643 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:01.370747 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:01.370757 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:01.370766 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:01.371013 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:01.371683 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:01.371688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:01.371692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:01.372365 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:01.372597 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:01.373484 (+ 887us) server_negotiation.cc:299] Negotiation successful
0928 03:09:01.373924 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2414,"thread_start_us":2256,"threads_started":1}
I20260928 03:09:01.374377 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:01.347807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:34019 (local address 127.6.192.2:36955)
0928 03:09:01.350537 (+ 2730us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:01.350556 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:01.350605 (+ 49us) server_negotiation.cc:407] Connection header received
0928 03:09:01.353002 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:01.353012 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:01.353519 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:01.353949 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:01.357289 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:01.358532 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:01.360518 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:01.361173 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:01.361398 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:01.366171 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:01.366182 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:01.366196 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:01.366245 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:01.370099 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:01.371008 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:01.371015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:01.371020 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:01.371268 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:01.371792 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:01.371796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:01.371798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:01.372375 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:01.372590 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:01.373497 (+ 907us) server_negotiation.cc:299] Negotiation successful
0928 03:09:01.373914 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2518,"thread_start_us":2338,"threads_started":1}
W20260928 03:09:01.377946 23646 leader_election.cc:341] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b85a9cf7cb14e0e9b734746e110f6f3 (127.6.192.3:43711): Not found: Tablet not found: b871e2cdc61d4245bf951e065020a1d9
W20260928 03:09:01.378235 23648 leader_election.cc:341] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955): Not found: Tablet not found: b871e2cdc61d4245bf951e065020a1d9
I20260928 03:09:01.378437 23648 leader_election.cc:302] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34c53cb0f3404a6aabaa805c6ada655a; no voters: 4322c37b370b4da3afb2b962da7f94d3, 7b85a9cf7cb14e0e9b734746e110f6f3
I20260928 03:09:01.378958 24028 raft_consensus.cc:2726] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:09:01.537024 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:01.536005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:47853)
0928 03:09:01.536568 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:01.536725 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":278,"threads_started":1}
I20260928 03:09:01.579705 24029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:01.579139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:56115)
0928 03:09:01.579341 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:01.579549 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 03:09:02.061106 24034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:02.060308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:56025)
0928 03:09:02.060822 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:02.060923 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":260,"threads_started":1}
I20260928 03:09:02.538879 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:02.538090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:52433)
0928 03:09:02.538620 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:02.538708 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":302,"threads_started":1}
I20260928 03:09:02.581666 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:02.580824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:44143)
0928 03:09:02.581362 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:02.581470 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":265,"threads_started":1}
I20260928 03:09:03.063131 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:03.062305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:41393)
0928 03:09:03.062781 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:03.062952 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":244,"threads_started":1}
I20260928 03:09:03.259963 24038 raft_consensus.cc:482] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:09:03.260245 24038 raft_consensus.cc:504] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } }
I20260928 03:09:03.261045 24038 leader_election.cc:288] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955), 7b85a9cf7cb14e0e9b734746e110f6f3 (127.6.192.3:43711)
W20260928 03:09:03.262365 23648 leader_election.cc:341] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955): Not found: Tablet not found: b871e2cdc61d4245bf951e065020a1d9
W20260928 03:09:03.262576 23646 leader_election.cc:341] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b85a9cf7cb14e0e9b734746e110f6f3 (127.6.192.3:43711): Not found: Tablet not found: b871e2cdc61d4245bf951e065020a1d9
I20260928 03:09:03.262707 23646 leader_election.cc:302] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34c53cb0f3404a6aabaa805c6ada655a; no voters: 4322c37b370b4da3afb2b962da7f94d3, 7b85a9cf7cb14e0e9b734746e110f6f3
I20260928 03:09:03.263017 24038 raft_consensus.cc:2726] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:09:03.540930 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:03.540045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:35835)
0928 03:09:03.540565 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:03.540667 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":283,"threads_started":1}
I20260928 03:09:03.583602 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:03.582654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:60931)
0928 03:09:03.583218 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:03.583320 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":312,"threads_started":1}
I20260928 03:09:04.065150 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:04.064356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:37793)
0928 03:09:04.064879 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:04.064962 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":237,"threads_started":1}
I20260928 03:09:04.542862 24042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:04.541963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:34009)
0928 03:09:04.542539 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:04.542652 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":275,"threads_started":1}
I20260928 03:09:04.585796 24043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:04.584934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:41783)
0928 03:09:04.585486 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:04.585600 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":276,"threads_started":1}
I20260928 03:09:05.067257 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:05.066443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:56259)
0928 03:09:05.066964 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:05.067067 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":266,"threads_started":1}
I20260928 03:09:05.540709 24045 raft_consensus.cc:482] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:09:05.540899 24045 raft_consensus.cc:504] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } }
I20260928 03:09:05.541674 24045 leader_election.cc:288] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955), 7b85a9cf7cb14e0e9b734746e110f6f3 (127.6.192.3:43711)
W20260928 03:09:05.542697 23646 leader_election.cc:341] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b85a9cf7cb14e0e9b734746e110f6f3 (127.6.192.3:43711): Not found: Tablet not found: b871e2cdc61d4245bf951e065020a1d9
W20260928 03:09:05.542923 23648 leader_election.cc:341] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955): Not found: Tablet not found: b871e2cdc61d4245bf951e065020a1d9
I20260928 03:09:05.543094 23648 leader_election.cc:302] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34c53cb0f3404a6aabaa805c6ada655a; no voters: 4322c37b370b4da3afb2b962da7f94d3, 7b85a9cf7cb14e0e9b734746e110f6f3
I20260928 03:09:05.543413 24045 raft_consensus.cc:2726] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:09:05.544521 24046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:05.543799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:60467)
0928 03:09:05.544292 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:05.544367 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":276,"threads_started":1}
I20260928 03:09:05.587924 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:05.586976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:47561)
0928 03:09:05.587539 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:05.587641 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":260,"threads_started":1}
I20260928 03:09:06.069329 24048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:06.068447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:44213)
0928 03:09:06.069014 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:06.069112 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":319,"threads_started":1}
I20260928 03:09:06.546219 24049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:06.545441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:33293)
0928 03:09:06.545964 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:06.546049 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":281,"threads_started":1}
I20260928 03:09:06.589973 24050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:06.589056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:34471)
0928 03:09:06.589652 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:06.589770 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":363,"threads_started":1}
I20260928 03:09:07.071214 24051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:07.070404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:53521)
0928 03:09:07.070932 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:07.071024 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":225,"threads_started":1}
I20260928 03:09:07.270610 24052 raft_consensus.cc:482] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:09:07.270797 24052 raft_consensus.cc:504] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } }
I20260928 03:09:07.271517 24052 leader_election.cc:288] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955), 7b85a9cf7cb14e0e9b734746e110f6f3 (127.6.192.3:43711)
W20260928 03:09:07.272445 23648 leader_election.cc:341] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955): Not found: Tablet not found: b871e2cdc61d4245bf951e065020a1d9
W20260928 03:09:07.272686 23646 leader_election.cc:341] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b85a9cf7cb14e0e9b734746e110f6f3 (127.6.192.3:43711): Not found: Tablet not found: b871e2cdc61d4245bf951e065020a1d9
I20260928 03:09:07.272820 23646 leader_election.cc:302] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34c53cb0f3404a6aabaa805c6ada655a; no voters: 4322c37b370b4da3afb2b962da7f94d3, 7b85a9cf7cb14e0e9b734746e110f6f3
I20260928 03:09:07.273132 24052 raft_consensus.cc:2726] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:09:07.548538 24053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:07.547297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:50687)
0928 03:09:07.548186 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:07.548307 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":488,"threads_started":1}
I20260928 03:09:07.591922 24054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:07.590989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:35857)
0928 03:09:07.591608 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:07.591701 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":265,"threads_started":1}
I20260928 03:09:08.073267 24055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:08.072374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:41347)
0928 03:09:08.072953 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:08.073071 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":240,"threads_started":1}
I20260928 03:09:08.550474 24056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:08.549696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:33017)
0928 03:09:08.550145 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:08.550232 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":239,"threads_started":1}
I20260928 03:09:08.593720 24057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:08.592870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:38783)
0928 03:09:08.593431 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:08.593519 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":268,"threads_started":1}
I20260928 03:09:09.075420 24058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:09.074444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:43349)
0928 03:09:09.075150 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:09.075244 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":374,"threads_started":1}
I20260928 03:09:09.552646 24059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:09.551675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:43939)
0928 03:09:09.552216 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:09.552463 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":273,"threads_started":1}
I20260928 03:09:09.595782 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:09.594942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:47341)
0928 03:09:09.595506 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:09.595616 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":262,"threads_started":1}
I20260928 03:09:10.077245 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:10.076483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:53853)
0928 03:09:10.076955 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:10.077047 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":240,"threads_started":1}
I20260928 03:09:10.381521 24062 raft_consensus.cc:482] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:09:10.381845 24062 raft_consensus.cc:504] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } }
I20260928 03:09:10.382681 24062 leader_election.cc:288] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955), 7b85a9cf7cb14e0e9b734746e110f6f3 (127.6.192.3:43711)
W20260928 03:09:10.383993 23648 leader_election.cc:341] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955): Not found: Tablet not found: b871e2cdc61d4245bf951e065020a1d9
W20260928 03:09:10.384196 23646 leader_election.cc:341] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b85a9cf7cb14e0e9b734746e110f6f3 (127.6.192.3:43711): Not found: Tablet not found: b871e2cdc61d4245bf951e065020a1d9
I20260928 03:09:10.384306 23646 leader_election.cc:302] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34c53cb0f3404a6aabaa805c6ada655a; no voters: 4322c37b370b4da3afb2b962da7f94d3, 7b85a9cf7cb14e0e9b734746e110f6f3
I20260928 03:09:10.384542 24062 raft_consensus.cc:2726] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:09:10.554598 24063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:10.553809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:54339)
0928 03:09:10.554302 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:10.554397 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":264,"threads_started":1}
I20260928 03:09:10.597880 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:10.596792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:33295)
0928 03:09:10.597389 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:10.597511 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":288,"threads_started":1}
I20260928 03:09:11.079272 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:11.078463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:44847)
0928 03:09:11.079012 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:11.079099 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":289,"threads_started":1}
I20260928 03:09:11.556629 24066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:11.555699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:37127)
0928 03:09:11.556363 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:11.556461 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":388,"threads_started":1}
I20260928 03:09:11.599843 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:11.598977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:44897)
0928 03:09:11.599538 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:11.599663 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":267,"threads_started":1}
I20260928 03:09:12.081147 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:12.080436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:57943)
0928 03:09:12.080855 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:12.080952 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":220,"threads_started":1}
I20260928 03:09:12.558652 24069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:12.557774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:37335)
0928 03:09:12.558377 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:12.558469 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":358,"threads_started":1}
I20260928 03:09:12.601852 24070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:12.601005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:59981)
0928 03:09:12.601572 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:12.601671 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":326,"threads_started":1}
I20260928 03:09:13.083087 24071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:13.082271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:46097)
0928 03:09:13.082748 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:13.082911 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":234,"threads_started":1}
I20260928 03:09:13.560611 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:13.559851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:45001)
0928 03:09:13.560330 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:13.560434 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":244,"threads_started":1}
I20260928 03:09:13.603830 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:13.603047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:41859)
0928 03:09:13.603565 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:13.603649 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":220,"threads_started":1}
I20260928 03:09:13.914057 24074 raft_consensus.cc:482] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:09:13.914311 24074 raft_consensus.cc:504] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c53cb0f3404a6aabaa805c6ada655a" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37375 } } peers { permanent_uuid: "4322c37b370b4da3afb2b962da7f94d3" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 36955 } } peers { permanent_uuid: "7b85a9cf7cb14e0e9b734746e110f6f3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43711 } }
I20260928 03:09:13.915015 24074 leader_election.cc:288] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955), 7b85a9cf7cb14e0e9b734746e110f6f3 (127.6.192.3:43711)
W20260928 03:09:13.916227 23646 leader_election.cc:341] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b85a9cf7cb14e0e9b734746e110f6f3 (127.6.192.3:43711): Not found: Tablet not found: b871e2cdc61d4245bf951e065020a1d9
W20260928 03:09:13.916435 23648 leader_election.cc:341] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4322c37b370b4da3afb2b962da7f94d3 (127.6.192.2:36955): Not found: Tablet not found: b871e2cdc61d4245bf951e065020a1d9
I20260928 03:09:13.916554 23648 leader_election.cc:302] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34c53cb0f3404a6aabaa805c6ada655a; no voters: 4322c37b370b4da3afb2b962da7f94d3, 7b85a9cf7cb14e0e9b734746e110f6f3
I20260928 03:09:13.916822 24074 raft_consensus.cc:2726] T b871e2cdc61d4245bf951e065020a1d9 P 34c53cb0f3404a6aabaa805c6ada655a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 03:09:14.084977 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:14.084114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:40667)
0928 03:09:14.084642 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:14.084737 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":286,"threads_started":1}
I20260928 03:09:14.562709 24076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:14.561724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:47401)
0928 03:09:14.562329 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:14.562418 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":320,"threads_started":1}
I20260928 03:09:14.606153 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:14.605313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.1:34445)
0928 03:09:14.605823 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:14.605949 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":250,"threads_started":1}
I20260928 03:09:15.087034 24078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:15.086224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.2:59993)
0928 03:09:15.086731 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:15.086832 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":261,"threads_started":1}
I20260928 03:09:15.537624 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 23589
I20260928 03:09:15.556666 24079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:15.539100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56034 (local address 127.6.192.1:37375)
0928 03:09:15.539808 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:15.539839 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:15.540232 (+ 393us) server_negotiation.cc:407] Connection header received
0928 03:09:15.540512 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:15.540520 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:15.540754 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:15.540913 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:15.543127 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:15.544379 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:15.545910 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:15.546594 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:15.546738 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:15.549962 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:15.549979 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:15.549995 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:15.550041 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:15.553306 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:15.554295 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:15.554304 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:15.554306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:15.554466 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:15.554919 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:15.554925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:15.554928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:15.555326 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:15.555637 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:15.556025 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 03:09:15.556283 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":358,"threads_started":1}
I20260928 03:09:15.558089 23755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:15.565091 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:15.564188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36919 (local address 127.6.192.3:51951)
0928 03:09:15.564768 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:15.564914 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.62:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":338,"threads_started":1}
I20260928 03:09:15.649565 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 23589
I20260928 03:09:15.664920 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 23763
I20260928 03:09:15.679222 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:15.665727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35072 (local address 127.6.192.2:36955)
0928 03:09:15.666350 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:15.666355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:15.666372 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:09:15.666430 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:15.666433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:15.666492 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:15.666583 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:15.668097 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:15.668893 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:15.670094 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:15.670503 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:15.670594 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:15.673714 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:15.673720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:15.673724 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:15.673760 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:15.676607 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:15.677301 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:15.677305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:15.677308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:15.677463 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:15.677929 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:15.677932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:15.677934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:15.678222 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:15.678443 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:15.678744 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 03:09:15.678919 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":399,"threads_started":1}
I20260928 03:09:15.680128 23886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:15.767853 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 23763
I20260928 03:09:15.785561 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 23894
I20260928 03:09:15.800246 24085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:15.786484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52438 (local address 127.6.192.3:43711)
0928 03:09:15.787066 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:15.787071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:15.787086 (+ 15us) server_negotiation.cc:407] Connection header received
0928 03:09:15.787141 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:15.787144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:15.787211 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:15.787338 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:15.788772 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:15.789524 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:15.790860 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:15.791327 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:15.791554 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:15.794278 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:15.794283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:15.794287 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:15.794324 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:15.797231 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:15.797980 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:15.797986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:15.797990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:15.798142 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:15.798555 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:15.798561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:15.798564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:15.798837 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:15.799121 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:15.799416 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 03:09:15.799928 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":323,"threads_started":1}
I20260928 03:09:15.801029 24017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:15.893244 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 23894
2026-09-28T03:09:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65020 ms)
[----------] 30 tests from SecurityITest (311216 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:09:16 dist-test-slave-fjww krb5kdc[24089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:09:16 dist-test-slave-fjww krb5kdc[24089](info): set up 2 sockets
Sep 28 03:09:16 dist-test-slave-fjww krb5kdc[24089](info): commencing operation
krb5kdc: starting...
W20260928 03:09:17.091948 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:09:17 dist-test-slave-fjww krb5kdc[24089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:09:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:09:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:09:17.372674 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:36153
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:36267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:36153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:17.714144 24102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:17.715093 24102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:17.715306 24102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:17.723984 24102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:09:17.724189 24102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:17.724323 24102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:17.724439 24102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:09:17.724525 24102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:09:17.736261 24102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:36267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:36153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:36153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:17.737833 24102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:17.739362 24102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:17.753919 24108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:17.753993 24110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:17.753971 24107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:17.755497 24102 server_base.cc:1034] running on GCE node
I20260928 03:09:17.756361 24102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:17.758082 24102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:17.759348 24102 hybrid_clock.cc:648] HybridClock initialized: now 1790564957759290 us; error 54 us; skew 500 ppm
I20260928 03:09:17.759783 24102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:17 dist-test-slave-fjww krb5kdc[24089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564957, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:17.763125 24102 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:09:17.764979 24102 webserver.cc:460] Webserver started at http://127.6.192.62:46579/ using document root <none> and password file <none>
I20260928 03:09:17.765730 24102 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:17.765893 24102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:17.766233 24102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:17.769536 24102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "fbc5dee94b164c0eae22b510b7865cb8"
format_stamp: "Formatted at 2026-09-28 03:09:17 on dist-test-slave-fjww"
server_key: "534367ff633de4b3145e6852584ead41"
server_key_iv: "a242b0220dc6ff32d771d986cdc04181"
server_key_version: "encryptionkey@0"
I20260928 03:09:17.770346 24102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "fbc5dee94b164c0eae22b510b7865cb8"
format_stamp: "Formatted at 2026-09-28 03:09:17 on dist-test-slave-fjww"
server_key: "534367ff633de4b3145e6852584ead41"
server_key_iv: "a242b0220dc6ff32d771d986cdc04181"
server_key_version: "encryptionkey@0"
I20260928 03:09:17.775988 24102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 03:09:17.779667 24117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:17.781219 24102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:09:17.781430 24102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "fbc5dee94b164c0eae22b510b7865cb8"
format_stamp: "Formatted at 2026-09-28 03:09:17 on dist-test-slave-fjww"
server_key: "534367ff633de4b3145e6852584ead41"
server_key_iv: "a242b0220dc6ff32d771d986cdc04181"
server_key_version: "encryptionkey@0"
I20260928 03:09:17.781663 24102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:17.822856 24102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:17.828111 24102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:17.828462 24102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:17.848815 24102 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:36153
I20260928 03:09:17.848929 24168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:36153 every 8 connection(s)
I20260928 03:09:17.850330 24102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:09:17.854653 24169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:17.860098 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 24102
I20260928 03:09:17.860347 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:09:17.861116 6912 external_mini_cluster.cc:1428] Setting key 79694dd54917ce993e7442787264876b
I20260928 03:09:17.867479 24169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8: Bootstrap starting.
I20260928 03:09:17.871596 24169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:17.873024 24169 log.cc:824] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:17.876927 24169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8: No bootstrap required, opened a new log
Sep 28 03:09:17 dist-test-slave-fjww krb5kdc[24089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:09:17.885720 24169 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc5dee94b164c0eae22b510b7865cb8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36153 } }
I20260928 03:09:17.886188 24169 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:17.886324 24169 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc5dee94b164c0eae22b510b7865cb8, State: Initialized, Role: FOLLOWER
I20260928 03:09:17.887041 24169 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc5dee94b164c0eae22b510b7865cb8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36153 } }
I20260928 03:09:17.887319 24169 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:09:17.887501 24169 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:09:17.887701 24169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:17.888039 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:17.863368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53050 (local address 127.6.192.62:36153)
0928 03:09:17.864458 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:17.864471 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:17.864517 (+ 46us) server_negotiation.cc:407] Connection header received
0928 03:09:17.866453 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:17.866480 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:17.866928 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:17.867289 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:17.869352 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:17.870820 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:17.872934 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:17.873929 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:17.874166 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:17.879403 (+ 5237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:17.879416 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:17.879434 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:17.879546 (+ 112us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:17.883145 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:17.884039 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:17.884047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:17.884054 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:17.884248 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:17.884682 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:17.884694 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:17.884697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:17.885407 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:17.885720 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:17.886212 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 03:09:17.886707 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":496,"threads_started":1}
I20260928 03:09:17.890205 24169 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc5dee94b164c0eae22b510b7865cb8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36153 } }
I20260928 03:09:17.890754 24169 leader_election.cc:302] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbc5dee94b164c0eae22b510b7865cb8; no voters:
I20260928 03:09:17.891466 24169 leader_election.cc:288] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:09:17.891614 24174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:17.892725 24174 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8 [term 1 LEADER]: Becoming Leader. State: Replica: fbc5dee94b164c0eae22b510b7865cb8, State: Running, Role: LEADER
I20260928 03:09:17.893391 24174 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc5dee94b164c0eae22b510b7865cb8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36153 } }
I20260928 03:09:17.893939 24169 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:09:17.897584 24175 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbc5dee94b164c0eae22b510b7865cb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc5dee94b164c0eae22b510b7865cb8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36153 } } }
I20260928 03:09:17.898166 24175 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8 [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:17.897246 24176 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbc5dee94b164c0eae22b510b7865cb8. Latest consensus state: current_term: 1 leader_uuid: "fbc5dee94b164c0eae22b510b7865cb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc5dee94b164c0eae22b510b7865cb8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36153 } } }
I20260928 03:09:17.899195 24176 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8 [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:17.901292 24181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:09:17.906857 24181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:09:17.916406 24181 catalog_manager.cc:1347] Generated new cluster ID: da75373a40db4982aab8324a369c4f41
I20260928 03:09:17.916543 24181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:09:17.939380 24181 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:09:17.940841 24181 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:09:17.952569 24181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8: Generated new TSK 0
I20260928 03:09:17.953657 24181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
I20260928 03:09:18.024528 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:36153
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:36267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:18.383713 24195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:18.384043 24195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:18.384227 24195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:18.392307 24195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:18.392473 24195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:18.392678 24195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:09:18.404477 24195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:36267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:36153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:18.406046 24195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:18.407529 24195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:18.423831 24200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:18.424988 24201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:18.429811 24203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:18.431236 24195 server_base.cc:1034] running on GCE node
I20260928 03:09:18.432281 24195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:18.433948 24195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:18.435204 24195 hybrid_clock.cc:648] HybridClock initialized: now 1790564958435133 us; error 77 us; skew 500 ppm
I20260928 03:09:18.435623 24195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:18 dist-test-slave-fjww krb5kdc[24089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564958, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:18.439136 24195 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:09:18.441092 24195 webserver.cc:460] Webserver started at http://127.6.192.1:36471/ using document root <none> and password file <none>
I20260928 03:09:18.441859 24195 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:18.442003 24195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:18.442340 24195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:18.445636 24195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "36c8b7b13c644d2d8ba9840822bca399"
format_stamp: "Formatted at 2026-09-28 03:09:18 on dist-test-slave-fjww"
server_key: "1c12a4d0ed3a2eea752891e35924366a"
server_key_iv: "da80449bf95c760c8c95c2a2e2154ec5"
server_key_version: "encryptionkey@0"
I20260928 03:09:18.446496 24195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "36c8b7b13c644d2d8ba9840822bca399"
format_stamp: "Formatted at 2026-09-28 03:09:18 on dist-test-slave-fjww"
server_key: "1c12a4d0ed3a2eea752891e35924366a"
server_key_iv: "da80449bf95c760c8c95c2a2e2154ec5"
server_key_version: "encryptionkey@0"
I20260928 03:09:18.452435 24195 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 03:09:18.456632 24210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:18.458169 24195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 03:09:18.458360 24195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "36c8b7b13c644d2d8ba9840822bca399"
format_stamp: "Formatted at 2026-09-28 03:09:18 on dist-test-slave-fjww"
server_key: "1c12a4d0ed3a2eea752891e35924366a"
server_key_iv: "da80449bf95c760c8c95c2a2e2154ec5"
server_key_version: "encryptionkey@0"
I20260928 03:09:18.458593 24195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:18.485579 24195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:18.490451 24195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:18.490777 24195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:18.491976 24195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:18.493472 24195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:18.493611 24195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:18.493757 24195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:18.493829 24195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:18.531344 24195 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:33109
I20260928 03:09:18.531512 24322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:33109 every 8 connection(s)
I20260928 03:09:18.532908 24195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:09:18.542922 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 24195
I20260928 03:09:18.543155 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:09:18.544037 6912 external_mini_cluster.cc:1428] Setting key 36388efac71004c05f02bbc9730e1c40
Sep 28 03:09:18 dist-test-slave-fjww krb5kdc[24089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564958, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:09:18.556664 24325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:18.535877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:45829 (local address 127.6.192.62:36153)
0928 03:09:18.536374 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:18.536409 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:18.537268 (+ 859us) server_negotiation.cc:407] Connection header received
0928 03:09:18.539305 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:18.539311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:18.539431 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:18.539612 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:18.541354 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:18.542092 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:18.544112 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:18.544658 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:18.544830 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:18.549101 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:18.549110 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:18.549118 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:18.549169 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:18.552967 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:18.553849 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:18.553859 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:18.553865 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:18.554028 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:18.554550 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:18.554557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:18.554563 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:18.555004 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:18.555162 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:18.556187 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0928 03:09:18.556386 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":259,"threads_started":1}
I20260928 03:09:18.558327 24326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:18.536344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36153 (local address 127.6.192.1:45829)
0928 03:09:18.537081 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:18.537124 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:18.538979 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:18.539903 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:18.539912 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:18.540499 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:18.541147 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:18.541167 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:18.542291 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:18.542295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:18.543926 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:18.543948 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:18.544938 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:18.544947 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:18.545290 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:18.546218 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:09:18.546253 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:18.548807 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:18.553151 (+ 4344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:18.553159 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:18.553175 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:18.553694 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:18.554189 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:18.554193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:18.554196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:18.554422 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:18.555167 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:18.555173 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:18.555719 (+ 546us) client_negotiation.cc:769] Sending connection context
0928 03:09:18.556067 (+ 348us) client_negotiation.cc:240] Negotiation successful
0928 03:09:18.556726 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":299,"threads_started":1}
I20260928 03:09:18.560070 24323 heartbeater.cc:347] Connected to a master server at 127.6.192.62:36153
I20260928 03:09:18.560560 24323 heartbeater.cc:464] Registering TS with master...
I20260928 03:09:18.561357 24323 heartbeater.cc:511] Master 127.6.192.62:36153 requested a full tablet report, sending...
I20260928 03:09:18.563968 24134 ts_manager.cc:194] Registered new tserver with Master: 36c8b7b13c644d2d8ba9840822bca399 (127.6.192.1:33109)
I20260928 03:09:18.565575 24134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:45829
WARNING: no policy specified for kudu/127.6.192.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
I20260928 03:09:18.606751 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:36153
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:36267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:18.951658 24329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:18.952005 24329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:18.952193 24329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:18.959928 24329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:18.960099 24329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:18.960283 24329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:09:18.972002 24329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:36267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:36153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:18.973496 24329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:18.975091 24329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:18.990991 24335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:18.990947 24334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:18.990921 24337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:18.992655 24329 server_base.cc:1034] running on GCE node
I20260928 03:09:18.993461 24329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:18.994695 24329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:18.995966 24329 hybrid_clock.cc:648] HybridClock initialized: now 1790564958995918 us; error 42 us; skew 500 ppm
I20260928 03:09:18.996358 24329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:18 dist-test-slave-fjww krb5kdc[24089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564958, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:19.000131 24329 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:09:19.002035 24329 webserver.cc:460] Webserver started at http://127.6.192.2:46749/ using document root <none> and password file <none>
I20260928 03:09:19.002967 24329 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:19.003118 24329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:19.003592 24329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:19.006837 24329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "eb6f60e68c1343f9b2271b4ce95435c1"
format_stamp: "Formatted at 2026-09-28 03:09:19 on dist-test-slave-fjww"
server_key: "4aa0622bd16165be62b073cfedcbf45c"
server_key_iv: "45a327c0def2491b71a70326c0143b60"
server_key_version: "encryptionkey@0"
I20260928 03:09:19.007833 24329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "eb6f60e68c1343f9b2271b4ce95435c1"
format_stamp: "Formatted at 2026-09-28 03:09:19 on dist-test-slave-fjww"
server_key: "4aa0622bd16165be62b073cfedcbf45c"
server_key_iv: "45a327c0def2491b71a70326c0143b60"
server_key_version: "encryptionkey@0"
I20260928 03:09:19.013885 24329 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 03:09:19.018180 24344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:19.020303 24329 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:09:19.020553 24329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "eb6f60e68c1343f9b2271b4ce95435c1"
format_stamp: "Formatted at 2026-09-28 03:09:19 on dist-test-slave-fjww"
server_key: "4aa0622bd16165be62b073cfedcbf45c"
server_key_iv: "45a327c0def2491b71a70326c0143b60"
server_key_version: "encryptionkey@0"
I20260928 03:09:19.020793 24329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:19.051247 24329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:19.056635 24329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:19.056958 24329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:19.058007 24329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:19.059640 24329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:19.059717 24329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:19.059859 24329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:19.059932 24329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:19.096715 24329 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:42431
I20260928 03:09:19.096757 24456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:42431 every 8 connection(s)
I20260928 03:09:19.098104 24329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:09:19.104138 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 24329
I20260928 03:09:19.104378 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:09:19.104983 6912 external_mini_cluster.cc:1428] Setting key 608a4801fb4b4f94489a59e5c7e1de76
Sep 28 03:09:19 dist-test-slave-fjww krb5kdc[24089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564958, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:09:19.122550 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:19.101081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:43983 (local address 127.6.192.62:36153)
0928 03:09:19.101552 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:19.101556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:19.102694 (+ 1138us) server_negotiation.cc:407] Connection header received
0928 03:09:19.105742 (+ 3048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:19.105749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:19.105854 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:19.105982 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:19.107867 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.108572 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.111205 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.111750 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.111851 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:19.116424 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:19.116430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:19.116433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:19.116469 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:19.119204 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:19.120104 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:19.120108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:19.120111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:19.120244 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:19.120717 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:19.120727 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:19.120729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:19.121006 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:19.121165 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:19.121906 (+ 741us) server_negotiation.cc:299] Negotiation successful
0928 03:09:19.122161 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":220,"threads_started":1}
I20260928 03:09:19.123739 24460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:19.101699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36153 (local address 127.6.192.2:43983)
0928 03:09:19.102505 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:19.102547 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:19.105386 (+ 2839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:19.106250 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:19.106260 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:19.106819 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:19.107640 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:19.107669 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.108757 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.108765 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:19.111026 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:19.111047 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.111992 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.112002 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:19.112352 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:19.113171 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:09:19.113206 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:19.116142 (+ 2936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:19.119388 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:19.119398 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:19.119417 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:19.119954 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:19.120363 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:19.120368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:19.120372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:19.120593 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:19.121144 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:19.121153 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:19.121534 (+ 381us) client_negotiation.cc:769] Sending connection context
0928 03:09:19.121801 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 03:09:19.122375 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":321,"threads_started":1}
I20260928 03:09:19.125299 24457 heartbeater.cc:347] Connected to a master server at 127.6.192.62:36153
I20260928 03:09:19.125665 24457 heartbeater.cc:464] Registering TS with master...
I20260928 03:09:19.126623 24457 heartbeater.cc:511] Master 127.6.192.62:36153 requested a full tablet report, sending...
I20260928 03:09:19.128319 24134 ts_manager.cc:194] Registered new tserver with Master: eb6f60e68c1343f9b2271b4ce95435c1 (127.6.192.2:42431)
I20260928 03:09:19.129187 24134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:43983
WARNING: no policy specified for kudu/127.6.192.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
I20260928 03:09:19.167364 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:36153
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:36267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:19.516100 24463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:19.516405 24463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:19.516561 24463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:19.524925 24463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:19.525068 24463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:19.525240 24463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:09:19.536257 24463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:36267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:36153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:19.537614 24463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:19.539008 24463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:19.553982 24468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:19.554000 24471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:19.554056 24469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:19.554795 24463 server_base.cc:1034] running on GCE node
I20260928 03:09:19.555600 24463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:19.556650 24463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:19.557978 24463 hybrid_clock.cc:648] HybridClock initialized: now 1790564959557930 us; error 42 us; skew 500 ppm
I20260928 03:09:19.558321 24463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:19 dist-test-slave-fjww krb5kdc[24089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564959, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:19.561817 24463 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:09:19.563616 24463 webserver.cc:460] Webserver started at http://127.6.192.3:44481/ using document root <none> and password file <none>
I20260928 03:09:19.564345 24463 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:19.564495 24463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:19.564821 24463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:19.568029 24463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "6faf9703fc234ebeb6d065522b183ff5"
format_stamp: "Formatted at 2026-09-28 03:09:19 on dist-test-slave-fjww"
server_key: "7d6d0db28ae5ce5d0caeca2cdacd0e68"
server_key_iv: "88897a2aa442c8c5c903324fd309035e"
server_key_version: "encryptionkey@0"
I20260928 03:09:19.568887 24463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "6faf9703fc234ebeb6d065522b183ff5"
format_stamp: "Formatted at 2026-09-28 03:09:19 on dist-test-slave-fjww"
server_key: "7d6d0db28ae5ce5d0caeca2cdacd0e68"
server_key_iv: "88897a2aa442c8c5c903324fd309035e"
server_key_version: "encryptionkey@0"
I20260928 03:09:19.568979 24323 heartbeater.cc:503] Master 127.6.192.62:36153 was elected leader, sending a full tablet report...
I20260928 03:09:19.574821 24463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 03:09:19.578665 24478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:19.580196 24463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:09:19.580392 24463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "6faf9703fc234ebeb6d065522b183ff5"
format_stamp: "Formatted at 2026-09-28 03:09:19 on dist-test-slave-fjww"
server_key: "7d6d0db28ae5ce5d0caeca2cdacd0e68"
server_key_iv: "88897a2aa442c8c5c903324fd309035e"
server_key_version: "encryptionkey@0"
I20260928 03:09:19.580615 24463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:19.602375 24463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:19.606871 24463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:19.607178 24463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:19.608289 24463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:19.609762 24463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:19.609975 24463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:19.610124 24463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:19.610200 24463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:19.646150 24463 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:35045
I20260928 03:09:19.646199 24590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:35045 every 8 connection(s)
I20260928 03:09:19.647547 24463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:09:19.654369 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 24463
I20260928 03:09:19.654604 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:09:19.655040 6912 external_mini_cluster.cc:1428] Setting key 57472798a0cfe4772684e006f0e72442
Sep 28 03:09:19 dist-test-slave-fjww krb5kdc[24089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564959, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:09:19.669821 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:19.650542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:57009 (local address 127.6.192.62:36153)
0928 03:09:19.651164 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:19.651168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:19.652053 (+ 885us) server_negotiation.cc:407] Connection header received
0928 03:09:19.654032 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:19.654038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:19.654137 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:19.654260 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:19.656207 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.657207 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.658906 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.659331 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.659488 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:19.663261 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:19.663273 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:19.663277 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:19.663315 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:19.666181 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:19.667073 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:19.667079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:19.667083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:19.667220 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:19.667775 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:19.667779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:19.667783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:19.668077 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:19.668250 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:19.669153 (+ 903us) server_negotiation.cc:299] Negotiation successful
0928 03:09:19.669407 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":391,"threads_started":1}
I20260928 03:09:19.671085 24594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:19.650989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36153 (local address 127.6.192.3:57009)
0928 03:09:19.651871 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:19.651927 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:19.653736 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:19.654543 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:19.654557 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:19.655177 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:19.656028 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:19.656050 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.657351 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.657357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:19.658755 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:19.658768 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.659581 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.659589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:19.659829 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:19.660482 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:09:19.660510 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:19.662971 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:19.666378 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:19.666389 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:19.666409 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:19.666891 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:19.667352 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:19.667357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:19.667362 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:19.667635 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:19.668215 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:19.668224 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:19.668695 (+ 471us) client_negotiation.cc:769] Sending connection context
0928 03:09:19.669058 (+ 363us) client_negotiation.cc:240] Negotiation successful
0928 03:09:19.669644 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":433,"threads_started":1}
I20260928 03:09:19.672669 24591 heartbeater.cc:347] Connected to a master server at 127.6.192.62:36153
I20260928 03:09:19.673046 24591 heartbeater.cc:464] Registering TS with master...
I20260928 03:09:19.673807 24591 heartbeater.cc:511] Master 127.6.192.62:36153 requested a full tablet report, sending...
I20260928 03:09:19.675303 24134 ts_manager.cc:194] Registered new tserver with Master: 6faf9703fc234ebeb6d065522b183ff5 (127.6.192.3:35045)
I20260928 03:09:19.676160 24134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:57009
I20260928 03:09:19.683907 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:09:19.702821 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:19.689032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53052 (local address 127.6.192.62:36153)
0928 03:09:19.689209 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:19.689213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:19.689617 (+ 404us) server_negotiation.cc:407] Connection header received
0928 03:09:19.689751 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:19.689755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:19.689822 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:19.689912 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:19.691495 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.692204 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.693567 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.694032 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.694134 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:19.696931 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:19.696936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:19.696939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:19.696974 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:19.699561 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:19.700516 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:19.700520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:19.700522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:19.700677 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:19.701199 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:19.701204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:19.701206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:19.701642 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:19.701890 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:19.702351 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 03:09:19.702556 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 03:09:19.708161 24134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:09:19.715312 24134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:09:19.740140 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:19.729757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:42431 (local address 127.0.0.1:55614)
0928 03:09:19.730366 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:19.730400 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:19.730515 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:19.731706 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:19.731711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:19.731737 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:19.732139 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:19.732155 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.734261 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.734267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:19.736480 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:19.736493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.738485 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.738492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:19.738887 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:19.739072 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 03:09:19.739207 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 03:09:19.739562 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":206,"thread_start_us":279,"threads_started":1}
I20260928 03:09:19.740165 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:19.728657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:33109 (local address 127.0.0.1:41834)
0928 03:09:19.729482 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:19.729531 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:19.729707 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:19.730575 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:19.730582 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:19.730615 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:19.731180 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:19.731194 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.733594 (+ 2400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.733600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:19.736316 (+ 2716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:19.736336 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.738500 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.738506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:19.738891 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:19.739072 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 03:09:19.739207 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 03:09:19.739563 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":230,"thread_start_us":361,"threads_started":1}
I20260928 03:09:19.741775 24605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:19.728963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41834 (local address 127.6.192.1:33109)
0928 03:09:19.729624 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:19.729633 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:19.729664 (+ 31us) server_negotiation.cc:407] Connection header received
0928 03:09:19.730014 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:19.730025 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:19.730326 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:19.731431 (+ 1105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:19.731728 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.733424 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.736560 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.738184 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.740032 (+ 1848us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:19.741136 (+ 1104us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:19.741232 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 03:09:19.741429 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":321,"threads_started":1}
I20260928 03:09:19.742117 24607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:19.730455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.6.192.2:42431)
0928 03:09:19.730991 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:19.731001 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:19.731035 (+ 34us) server_negotiation.cc:407] Connection header received
0928 03:09:19.731137 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:19.731144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:19.731459 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:19.732345 (+ 886us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:19.732572 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.734107 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.736681 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.738177 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.741088 (+ 2911us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:19.741343 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:19.741423 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 03:09:19.741597 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":493,"threads_started":1}
I20260928 03:09:19.746183 24258 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9f692483d645faa7a82006751ed47a (DEFAULT_TABLE table=test-table [id=a3c4a2a1e38b4c1582868aa7214cc173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:09:19.746196 24392 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9f692483d645faa7a82006751ed47a (DEFAULT_TABLE table=test-table [id=a3c4a2a1e38b4c1582868aa7214cc173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:09:19.747882 24392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a9f692483d645faa7a82006751ed47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:19.747891 24258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a9f692483d645faa7a82006751ed47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:19.750821 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:19.731145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:35045 (local address 127.0.0.1:46214)
0928 03:09:19.740767 (+ 9622us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:19.740788 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:19.740920 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:19.744814 (+ 3894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:19.744820 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:19.744859 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:19.745059 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:19.745075 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.746468 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.746473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:19.748663 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:19.748677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.749951 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.749957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:19.750158 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:19.750196 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 03:09:19.750318 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 03:09:19.750487 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9503,"thread_start_us":12021,"threads_started":1}
I20260928 03:09:19.750859 24609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:19.731631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46214 (local address 127.6.192.3:35045)
0928 03:09:19.744251 (+ 12620us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:19.744258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:19.744288 (+ 30us) server_negotiation.cc:407] Connection header received
0928 03:09:19.744385 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:19.744390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:19.744593 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:19.744829 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:19.745242 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.746298 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.748832 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.749782 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.749938 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:19.750188 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:19.750372 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 03:09:19.750529 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12439,"thread_start_us":12287,"threads_started":1}
I20260928 03:09:19.754432 24526 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9f692483d645faa7a82006751ed47a (DEFAULT_TABLE table=test-table [id=a3c4a2a1e38b4c1582868aa7214cc173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:09:19.755966 24526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a9f692483d645faa7a82006751ed47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:19.764181 24611 tablet_bootstrap.cc:492] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1: Bootstrap starting.
I20260928 03:09:19.764626 24610 tablet_bootstrap.cc:492] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399: Bootstrap starting.
I20260928 03:09:19.768636 24611 tablet_bootstrap.cc:654] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:19.769157 24612 tablet_bootstrap.cc:492] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5: Bootstrap starting.
I20260928 03:09:19.771260 24610 tablet_bootstrap.cc:654] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:19.771528 24611 log.cc:824] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:19.772680 24610 log.cc:824] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:19.773011 24612 tablet_bootstrap.cc:654] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:19.774247 24612 log.cc:824] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:19.774943 24611 tablet_bootstrap.cc:492] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1: No bootstrap required, opened a new log
I20260928 03:09:19.775278 24611 ts_tablet_manager.cc:1397] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260928 03:09:19.775966 24610 tablet_bootstrap.cc:492] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399: No bootstrap required, opened a new log
I20260928 03:09:19.776257 24610 ts_tablet_manager.cc:1397] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260928 03:09:19.777436 24612 tablet_bootstrap.cc:492] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5: No bootstrap required, opened a new log
I20260928 03:09:19.777765 24612 ts_tablet_manager.cc:1397] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 03:09:19.784072 24611 raft_consensus.cc:348] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c8b7b13c644d2d8ba9840822bca399" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33109 } } peers { permanent_uuid: "eb6f60e68c1343f9b2271b4ce95435c1" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42431 } } peers { permanent_uuid: "6faf9703fc234ebeb6d065522b183ff5" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35045 } }
I20260928 03:09:19.784598 24611 raft_consensus.cc:374] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:19.784708 24611 raft_consensus.cc:729] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6f60e68c1343f9b2271b4ce95435c1, State: Initialized, Role: FOLLOWER
I20260928 03:09:19.785712 24611 consensus_queue.cc:260] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c8b7b13c644d2d8ba9840822bca399" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33109 } } peers { permanent_uuid: "eb6f60e68c1343f9b2271b4ce95435c1" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42431 } } peers { permanent_uuid: "6faf9703fc234ebeb6d065522b183ff5" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35045 } }
I20260928 03:09:19.787999 24610 raft_consensus.cc:348] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c8b7b13c644d2d8ba9840822bca399" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33109 } } peers { permanent_uuid: "eb6f60e68c1343f9b2271b4ce95435c1" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42431 } } peers { permanent_uuid: "6faf9703fc234ebeb6d065522b183ff5" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35045 } }
I20260928 03:09:19.788496 24610 raft_consensus.cc:374] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:19.788615 24610 raft_consensus.cc:729] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36c8b7b13c644d2d8ba9840822bca399, State: Initialized, Role: FOLLOWER
I20260928 03:09:19.788938 24612 raft_consensus.cc:348] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c8b7b13c644d2d8ba9840822bca399" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33109 } } peers { permanent_uuid: "eb6f60e68c1343f9b2271b4ce95435c1" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42431 } } peers { permanent_uuid: "6faf9703fc234ebeb6d065522b183ff5" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35045 } }
I20260928 03:09:19.789402 24612 raft_consensus.cc:374] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:19.789505 24612 raft_consensus.cc:729] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6faf9703fc234ebeb6d065522b183ff5, State: Initialized, Role: FOLLOWER
I20260928 03:09:19.789320 24610 consensus_queue.cc:260] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c8b7b13c644d2d8ba9840822bca399" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33109 } } peers { permanent_uuid: "eb6f60e68c1343f9b2271b4ce95435c1" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42431 } } peers { permanent_uuid: "6faf9703fc234ebeb6d065522b183ff5" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35045 } }
I20260928 03:09:19.790192 24612 consensus_queue.cc:260] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c8b7b13c644d2d8ba9840822bca399" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33109 } } peers { permanent_uuid: "eb6f60e68c1343f9b2271b4ce95435c1" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42431 } } peers { permanent_uuid: "6faf9703fc234ebeb6d065522b183ff5" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35045 } }
I20260928 03:09:19.791819 24591 heartbeater.cc:503] Master 127.6.192.62:36153 was elected leader, sending a full tablet report...
I20260928 03:09:19.792964 24457 heartbeater.cc:503] Master 127.6.192.62:36153 was elected leader, sending a full tablet report...
I20260928 03:09:19.794131 24611 ts_tablet_manager.cc:1428] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1: Time spent starting tablet: real 0.019s user 0.015s sys 0.004s
I20260928 03:09:19.797168 24612 ts_tablet_manager.cc:1428] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5: Time spent starting tablet: real 0.019s user 0.015s sys 0.000s
I20260928 03:09:19.798966 24610 ts_tablet_manager.cc:1428] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399: Time spent starting tablet: real 0.023s user 0.018s sys 0.004s
I20260928 03:09:19.807312 24616 raft_consensus.cc:482] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:09:19.807682 24616 raft_consensus.cc:504] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c8b7b13c644d2d8ba9840822bca399" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33109 } } peers { permanent_uuid: "eb6f60e68c1343f9b2271b4ce95435c1" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42431 } } peers { permanent_uuid: "6faf9703fc234ebeb6d065522b183ff5" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35045 } }
I20260928 03:09:19.809517 24616 leader_election.cc:288] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36c8b7b13c644d2d8ba9840822bca399 (127.6.192.1:33109), 6faf9703fc234ebeb6d065522b183ff5 (127.6.192.3:35045)
I20260928 03:09:19.818801 24605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:19.809998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:41689 (local address 127.6.192.1:33109)
0928 03:09:19.810216 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:19.810224 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:19.810896 (+ 672us) server_negotiation.cc:407] Connection header received
0928 03:09:19.811261 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:19.811271 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:19.811380 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:19.811564 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:19.812159 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.813165 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.816003 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.817329 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.817501 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:19.817636 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:19.818385 (+ 749us) server_negotiation.cc:299] Negotiation successful
0928 03:09:19.818531 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 03:09:19.819048 24619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:19.809868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:33109 (local address 127.6.192.2:41689)
0928 03:09:19.810772 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:19.810793 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:19.811041 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:19.811579 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:19.811583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:19.811611 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:19.812015 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:19.812031 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.813325 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.813329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:19.815789 (+ 2460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:19.815805 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.817652 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.817662 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:19.818106 (+ 444us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:19.818205 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 03:09:19.818315 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 03:09:19.818502 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":212,"thread_start_us":568,"threads_started":1}
I20260928 03:09:19.819092 24620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:19.810130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:35045 (local address 127.6.192.2:60319)
0928 03:09:19.811230 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:19.811251 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:19.811507 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:19.812022 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:19.812025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:19.812082 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:19.812548 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:19.812560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.813898 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.813903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:19.816009 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:19.816019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.817690 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.817695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:19.818103 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:19.818205 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 03:09:19.818313 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 03:09:19.818502 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"mutex_wait_us":125,"thread_start_us":720,"threads_started":1}
I20260928 03:09:19.818796 24609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:19.810094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:60319 (local address 127.6.192.3:35045)
0928 03:09:19.810282 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:19.810289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:19.811350 (+ 1061us) server_negotiation.cc:407] Connection header received
0928 03:09:19.811686 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:19.811692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:19.811784 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:19.811843 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:19.812712 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.813634 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.816156 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.817484 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.817599 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:19.817702 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:19.818385 (+ 683us) server_negotiation.cc:299] Negotiation successful
0928 03:09:19.818531 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 03:09:19.820158 24278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a9f692483d645faa7a82006751ed47a" candidate_uuid: "eb6f60e68c1343f9b2271b4ce95435c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36c8b7b13c644d2d8ba9840822bca399" is_pre_election: true
I20260928 03:09:19.820180 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a9f692483d645faa7a82006751ed47a" candidate_uuid: "eb6f60e68c1343f9b2271b4ce95435c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6faf9703fc234ebeb6d065522b183ff5" is_pre_election: true
I20260928 03:09:19.820709 24546 raft_consensus.cc:2454] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6f60e68c1343f9b2271b4ce95435c1 in term 0.
I20260928 03:09:19.820709 24278 raft_consensus.cc:2454] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6f60e68c1343f9b2271b4ce95435c1 in term 0.
I20260928 03:09:19.821467 24348 leader_election.cc:302] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36c8b7b13c644d2d8ba9840822bca399, eb6f60e68c1343f9b2271b4ce95435c1; no voters:
I20260928 03:09:19.821894 24616 raft_consensus.cc:2781] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:09:19.822057 24616 raft_consensus.cc:482] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:09:19.822163 24616 raft_consensus.cc:3037] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:19.824635 24616 raft_consensus.cc:504] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c8b7b13c644d2d8ba9840822bca399" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33109 } } peers { permanent_uuid: "eb6f60e68c1343f9b2271b4ce95435c1" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42431 } } peers { permanent_uuid: "6faf9703fc234ebeb6d065522b183ff5" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35045 } }
I20260928 03:09:19.825312 24616 leader_election.cc:288] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [CANDIDATE]: Term 1 election: Requested vote from peers 36c8b7b13c644d2d8ba9840822bca399 (127.6.192.1:33109), 6faf9703fc234ebeb6d065522b183ff5 (127.6.192.3:35045)
I20260928 03:09:19.825941 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a9f692483d645faa7a82006751ed47a" candidate_uuid: "eb6f60e68c1343f9b2271b4ce95435c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6faf9703fc234ebeb6d065522b183ff5"
I20260928 03:09:19.826174 24546 raft_consensus.cc:3037] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:19.826159 24278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a9f692483d645faa7a82006751ed47a" candidate_uuid: "eb6f60e68c1343f9b2271b4ce95435c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36c8b7b13c644d2d8ba9840822bca399"
I20260928 03:09:19.826434 24278 raft_consensus.cc:3037] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:19.828732 24546 raft_consensus.cc:2454] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6f60e68c1343f9b2271b4ce95435c1 in term 1.
I20260928 03:09:19.829180 24278 raft_consensus.cc:2454] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6f60e68c1343f9b2271b4ce95435c1 in term 1.
I20260928 03:09:19.829319 24347 leader_election.cc:302] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6faf9703fc234ebeb6d065522b183ff5, eb6f60e68c1343f9b2271b4ce95435c1; no voters:
I20260928 03:09:19.829772 24616 raft_consensus.cc:2781] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:19.830259 24616 raft_consensus.cc:686] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [term 1 LEADER]: Becoming Leader. State: Replica: eb6f60e68c1343f9b2271b4ce95435c1, State: Running, Role: LEADER
I20260928 03:09:19.830917 24616 consensus_queue.cc:237] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c8b7b13c644d2d8ba9840822bca399" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33109 } } peers { permanent_uuid: "eb6f60e68c1343f9b2271b4ce95435c1" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42431 } } peers { permanent_uuid: "6faf9703fc234ebeb6d065522b183ff5" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35045 } }
I20260928 03:09:19.836016 24132 catalog_manager.cc:5495] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb6f60e68c1343f9b2271b4ce95435c1 (127.6.192.2). New cstate: current_term: 1 leader_uuid: "eb6f60e68c1343f9b2271b4ce95435c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c8b7b13c644d2d8ba9840822bca399" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6f60e68c1343f9b2271b4ce95435c1" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6faf9703fc234ebeb6d065522b183ff5" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35045 } health_report { overall_health: UNKNOWN } } }
W20260928 03:09:19.850250 24458 tablet.cc:2369] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:09:19.871632 24607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:19.865793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55630 (local address 127.6.192.2:42431)
0928 03:09:19.866040 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:19.866046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:19.866065 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:09:19.866286 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:19.866292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:19.866397 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:19.866559 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:09:19.866972 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.867796 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.869249 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.869660 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.869810 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:19.870342 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:09:19.870906 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:09:19.871080 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:19.871193 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 03:09:19.871283 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 03:09:19.880642 24546 raft_consensus.cc:1261] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5 [term 1 FOLLOWER]: Refusing update from remote peer eb6f60e68c1343f9b2271b4ce95435c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:09:19.880694 24278 raft_consensus.cc:1261] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399 [term 1 FOLLOWER]: Refusing update from remote peer eb6f60e68c1343f9b2271b4ce95435c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:09:19.881835 24621 consensus_queue.cc:1035] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36c8b7b13c644d2d8ba9840822bca399" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:09:19.882364 24616 consensus_queue.cc:1035] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6faf9703fc234ebeb6d065522b183ff5" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 35045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 03:09:19.899403 24592 tablet.cc:2369] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:09:19.900923 24626 mvcc.cc:205] Tried to move back new op lower bound from 7334154075659059200 to 7334154075472613376. Current Snapshot: MvccSnapshot[applied={T|T < 7334154075659059200}]
I20260928 03:09:19.901682 24624 mvcc.cc:205] Tried to move back new op lower bound from 7334154075659059200 to 7334154075472613376. Current Snapshot: MvccSnapshot[applied={T|T < 7334154075659059200}]
I20260928 03:09:19.915223 24134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:09:19.915570 24134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:09:19.919687 24134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fbc5dee94b164c0eae22b510b7865cb8: Sending DeleteTablet for 3 replicas of tablet 9a9f692483d645faa7a82006751ed47a
I20260928 03:09:19.920881 24258 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a9f692483d645faa7a82006751ed47a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:09:19 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:41834
I20260928 03:09:19.921025 24392 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a9f692483d645faa7a82006751ed47a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:09:19 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:55614
I20260928 03:09:19.921758 24526 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a9f692483d645faa7a82006751ed47a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:09:19 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:46214
I20260928 03:09:19.922451 24634 tablet_replica.cc:330] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399: stopping tablet replica
I20260928 03:09:19.922616 24635 tablet_replica.cc:330] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1: stopping tablet replica
I20260928 03:09:19.922971 24635 raft_consensus.cc:2229] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:09:19.922974 24634 raft_consensus.cc:2229] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:09:19.923287 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 24195
I20260928 03:09:19.923620 24634 raft_consensus.cc:2258] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:09:19.923715 24635 raft_consensus.cc:2258] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:09:19.926009 24635 ts_tablet_manager.cc:1905] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:09:19.930442 24634 ts_tablet_manager.cc:1905] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:09:19.934739 24635 ts_tablet_manager.cc:1918] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:09:19.934948 24635 log.cc:1196] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/9a9f692483d645faa7a82006751ed47a
I20260928 03:09:19.935567 24636 tablet_replica.cc:330] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5: stopping tablet replica
I20260928 03:09:19.935635 24635 ts_tablet_manager.cc:1939] T 9a9f692483d645faa7a82006751ed47a P eb6f60e68c1343f9b2271b4ce95435c1: Deleting consensus metadata
I20260928 03:09:19.936233 24636 raft_consensus.cc:2229] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:09:19.936906 24636 raft_consensus.cc:2258] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:09:19.937773 24118 catalog_manager.cc:4841] TS eb6f60e68c1343f9b2271b4ce95435c1 (127.6.192.2:42431): tablet 9a9f692483d645faa7a82006751ed47a (table test-table [id=a3c4a2a1e38b4c1582868aa7214cc173]) successfully deleted
I20260928 03:09:19.939540 24636 ts_tablet_manager.cc:1905] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:09:19.942485 24634 ts_tablet_manager.cc:1918] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:09:19.942685 24634 log.cc:1196] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/9a9f692483d645faa7a82006751ed47a
I20260928 03:09:19.943262 24634 ts_tablet_manager.cc:1939] T 9a9f692483d645faa7a82006751ed47a P 36c8b7b13c644d2d8ba9840822bca399: Deleting consensus metadata
I20260928 03:09:19.945390 24121 catalog_manager.cc:4841] TS 36c8b7b13c644d2d8ba9840822bca399 (127.6.192.1:33109): tablet 9a9f692483d645faa7a82006751ed47a (table test-table [id=a3c4a2a1e38b4c1582868aa7214cc173]) successfully deleted
I20260928 03:09:19.946691 24636 ts_tablet_manager.cc:1918] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:09:19.946902 24636 log.cc:1196] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/9a9f692483d645faa7a82006751ed47a
I20260928 03:09:19.947422 24636 ts_tablet_manager.cc:1939] T 9a9f692483d645faa7a82006751ed47a P 6faf9703fc234ebeb6d065522b183ff5: Deleting consensus metadata
I20260928 03:09:19.949558 24120 catalog_manager.cc:4841] TS 6faf9703fc234ebeb6d065522b183ff5 (127.6.192.3:35045): tablet 9a9f692483d645faa7a82006751ed47a (table test-table [id=a3c4a2a1e38b4c1582868aa7214cc173]) successfully deleted
Sep 28 03:09:19 dist-test-slave-fjww krb5kdc[24089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:09:19.964625 24605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:19.924649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41840 (local address 127.6.192.1:33109)
0928 03:09:19.924878 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:19.924887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:19.935985 (+ 11098us) server_negotiation.cc:407] Connection header received
0928 03:09:19.936557 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:19.936564 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:19.936652 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:19.936722 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:19.938639 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.940124 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.945267 (+ 5143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:19.946031 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:19.946193 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:19.957178 (+ 10985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:19.957199 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:19.957215 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:19.957263 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:19.960654 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:19.961939 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:19.961948 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:19.961950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:19.962130 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:19.962676 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:19.962680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:19.962682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:19.963281 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:19.963676 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:19.964154 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 03:09:19.964342 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 03:09:19.965560 24318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:20.060992 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 24195
I20260928 03:09:20.087503 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 24329
Sep 28 03:09:20 dist-test-slave-fjww krb5kdc[24089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.2@KRBTEST.COM
I20260928 03:09:20.110491 24607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:20.088496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55642 (local address 127.6.192.2:42431)
0928 03:09:20.088730 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:20.088737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:20.088761 (+ 24us) server_negotiation.cc:407] Connection header received
0928 03:09:20.088871 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:20.088877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:20.088966 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:20.089093 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:20.090454 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:20.092024 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:20.093566 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:20.094428 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:20.094576 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:20.101409 (+ 6833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:20.101427 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:20.101448 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:20.101519 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:20.106543 (+ 5024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:20.107709 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:20.107721 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:20.107726 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:20.107953 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:20.108503 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:20.108509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:20.108514 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:20.109045 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:20.109458 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:20.109787 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 03:09:20.110048 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 03:09:20.111323 24452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:20.228168 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 24329
I20260928 03:09:20.248262 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 24463
Sep 28 03:09:20 dist-test-slave-fjww krb5kdc[24089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.3@KRBTEST.COM
I20260928 03:09:20.265506 24609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:20.249197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46218 (local address 127.6.192.3:35045)
0928 03:09:20.249435 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:20.249442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:20.249464 (+ 22us) server_negotiation.cc:407] Connection header received
0928 03:09:20.249663 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:20.249667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:20.249732 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:20.249839 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:20.251083 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:20.251908 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:20.253240 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:20.253661 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:20.253804 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:20.258782 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:20.258800 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:20.258817 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:20.258867 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:20.261985 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:20.262866 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:20.262875 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:20.262883 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:20.263049 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:20.263770 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:20.263777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:20.263781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:20.264198 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:20.264521 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:20.264914 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 03:09:20.265160 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 03:09:20.266294 24586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:20.359056 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 24463
I20260928 03:09:20.375710 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 24102
I20260928 03:09:20.377038 24164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:20.488515 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 24102
2026-09-28T03:09:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4598 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T03:09:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:09:20Z Disabled control of system clock
I20260928 03:09:20.573400 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:38817
--webserver_interface=127.6.192.62
--webserver_port=0
--builtin_ntp_servers=127.6.192.20:38565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:38817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 03:09:20.921941 24645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:20.922250 24645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:20.922341 24645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:20.930083 24645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:09:20.930213 24645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:20.930253 24645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:20.930303 24645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:09:20.930348 24645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:09:20.943311 24645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:38817
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:38817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:20.944931 24645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:20.946790 24645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:20.961596 24650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:20.961658 24653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:20.961628 24651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:20.962386 24645 server_base.cc:1034] running on GCE node
I20260928 03:09:20.963186 24645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:20.964777 24645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:20.966001 24645 hybrid_clock.cc:648] HybridClock initialized: now 1790564960965957 us; error 46 us; skew 500 ppm
I20260928 03:09:20.966421 24645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:09:20.969702 24645 webserver.cc:460] Webserver started at http://127.6.192.62:45769/ using document root <none> and password file <none>
I20260928 03:09:20.970539 24645 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:20.970635 24645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:20.970928 24645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:20.974329 24645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "1ea479b044f44169afd6233496a55889"
format_stamp: "Formatted at 2026-09-28 03:09:20 on dist-test-slave-fjww"
server_key: "37748ef93bd33d84fda49ba1464aca78"
server_key_iv: "1567735f6d29844aa4641dc6a61653d4"
server_key_version: "encryptionkey@0"
I20260928 03:09:20.975364 24645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "1ea479b044f44169afd6233496a55889"
format_stamp: "Formatted at 2026-09-28 03:09:20 on dist-test-slave-fjww"
server_key: "37748ef93bd33d84fda49ba1464aca78"
server_key_iv: "1567735f6d29844aa4641dc6a61653d4"
server_key_version: "encryptionkey@0"
I20260928 03:09:20.981590 24645 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 03:09:20.985893 24659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:20.987542 24645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 03:09:20.987769 24645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "1ea479b044f44169afd6233496a55889"
format_stamp: "Formatted at 2026-09-28 03:09:20 on dist-test-slave-fjww"
server_key: "37748ef93bd33d84fda49ba1464aca78"
server_key_iv: "1567735f6d29844aa4641dc6a61653d4"
server_key_version: "encryptionkey@0"
I20260928 03:09:20.988106 24645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:21.015159 24645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:21.016251 24645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:21.016615 24645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:21.038336 24645 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:38817
I20260928 03:09:21.038357 24710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:38817 every 8 connection(s)
I20260928 03:09:21.040055 24645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:09:21.044185 24711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:21.049283 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 24645
I20260928 03:09:21.049489 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:09:21.050099 6912 external_mini_cluster.cc:1428] Setting key 1d5ea4d311f917aed78eb18b6c60e052
I20260928 03:09:21.057164 24711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889: Bootstrap starting.
I20260928 03:09:21.061309 24711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:21.062700 24711 log.cc:824] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:21.066676 24711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889: No bootstrap required, opened a new log
I20260928 03:09:21.067335 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:21.052424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52796 (local address 127.6.192.62:38817)
0928 03:09:21.053615 (+ 1191us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:21.053630 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:21.053686 (+ 56us) server_negotiation.cc:407] Connection header received
0928 03:09:21.055849 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:21.055878 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:21.056459 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:21.056931 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:21.057566 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:21.059086 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:21.061268 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:21.062544 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:21.062765 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:21.064445 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:21.064458 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:21.064485 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:21.065152 (+ 667us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:21.065253 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:21.065280 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:21.065415 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:21.065741 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 03:09:21.066173 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":429,"threads_started":1}
I20260928 03:09:21.075098 24711 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea479b044f44169afd6233496a55889" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38817 } }
I20260928 03:09:21.075639 24711 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:21.075735 24711 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea479b044f44169afd6233496a55889, State: Initialized, Role: FOLLOWER
I20260928 03:09:21.076277 24711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea479b044f44169afd6233496a55889" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38817 } }
I20260928 03:09:21.076493 24711 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:09:21.076596 24711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:09:21.076726 24711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:21.078975 24711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea479b044f44169afd6233496a55889" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38817 } }
I20260928 03:09:21.079571 24711 leader_election.cc:302] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ea479b044f44169afd6233496a55889; no voters:
I20260928 03:09:21.080217 24711 leader_election.cc:288] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:09:21.080467 24716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:21.081279 24716 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889 [term 1 LEADER]: Becoming Leader. State: Replica: 1ea479b044f44169afd6233496a55889, State: Running, Role: LEADER
I20260928 03:09:21.081928 24716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea479b044f44169afd6233496a55889" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38817 } }
I20260928 03:09:21.082523 24711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:09:21.085304 24718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ea479b044f44169afd6233496a55889. Latest consensus state: current_term: 1 leader_uuid: "1ea479b044f44169afd6233496a55889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea479b044f44169afd6233496a55889" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38817 } } }
I20260928 03:09:21.085788 24718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889 [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:21.085628 24717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ea479b044f44169afd6233496a55889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea479b044f44169afd6233496a55889" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38817 } } }
I20260928 03:09:21.086043 24717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889 [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:21.088860 24725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:09:21.094887 24725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:09:21.105675 24725 catalog_manager.cc:1347] Generated new cluster ID: d95780e563cf4cc3b6f5be73c58712c0
I20260928 03:09:21.105831 24725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:09:21.123292 24725 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:09:21.125433 24725 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:09:21.142737 24725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889: Generated new TSK 0
I20260928 03:09:21.144083 24725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:09:21.159432 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:38817
--builtin_ntp_servers=127.6.192.20:38565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 03:09:21.509519 24735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:21.509824 24735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:21.509958 24735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:21.517536 24735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:21.517652 24735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:21.517835 24735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:09:21.527494 24735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:38817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:21.528838 24735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:21.530721 24735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:21.544685 24740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:21.544835 24743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:21.544723 24741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:21.546079 24735 server_base.cc:1034] running on GCE node
I20260928 03:09:21.546783 24735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:21.547969 24735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:21.549187 24735 hybrid_clock.cc:648] HybridClock initialized: now 1790564961549113 us; error 66 us; skew 500 ppm
I20260928 03:09:21.549540 24735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:09:21.552330 24735 webserver.cc:460] Webserver started at http://127.6.192.1:43761/ using document root <none> and password file <none>
I20260928 03:09:21.553102 24735 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:21.553200 24735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:21.553576 24735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:21.556978 24735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "29f4a10c98714ef8a5075114b6236cf4"
format_stamp: "Formatted at 2026-09-28 03:09:21 on dist-test-slave-fjww"
server_key: "dd4e36f759ac5059d4c3b2669c46bcca"
server_key_iv: "e5df8244876d08cfd4dc54828cc10e5b"
server_key_version: "encryptionkey@0"
I20260928 03:09:21.557973 24735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "29f4a10c98714ef8a5075114b6236cf4"
format_stamp: "Formatted at 2026-09-28 03:09:21 on dist-test-slave-fjww"
server_key: "dd4e36f759ac5059d4c3b2669c46bcca"
server_key_iv: "e5df8244876d08cfd4dc54828cc10e5b"
server_key_version: "encryptionkey@0"
I20260928 03:09:21.563912 24735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 03:09:21.568161 24749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:21.569799 24735 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 03:09:21.569970 24735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "29f4a10c98714ef8a5075114b6236cf4"
format_stamp: "Formatted at 2026-09-28 03:09:21 on dist-test-slave-fjww"
server_key: "dd4e36f759ac5059d4c3b2669c46bcca"
server_key_iv: "e5df8244876d08cfd4dc54828cc10e5b"
server_key_version: "encryptionkey@0"
I20260928 03:09:21.570254 24735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:21.591713 24735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:21.592638 24735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:21.592964 24735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:21.594081 24735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:21.595674 24735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:21.595762 24735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:21.595923 24735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:21.596022 24735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:21.630980 24735 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:45411
I20260928 03:09:21.631102 24861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:45411 every 8 connection(s)
I20260928 03:09:21.632401 24735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:09:21.635965 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 24735
I20260928 03:09:21.636158 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:09:21.636871 6912 external_mini_cluster.cc:1428] Setting key f7641cdd73867a73fee9984cb66c96e0
I20260928 03:09:21.640200 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:38817
--builtin_ntp_servers=127.6.192.20:38565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 03:09:21.648636 24864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:21.635303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:50489 (local address 127.6.192.62:38817)
0928 03:09:21.635882 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:21.635890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:21.637592 (+ 1702us) server_negotiation.cc:407] Connection header received
0928 03:09:21.639799 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:21.639803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:21.639870 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:21.639972 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:21.641615 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:21.642490 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:21.645323 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:21.645796 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:21.645991 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:21.647815 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:21.647820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:21.647824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:21.647844 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:21.647904 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:21.647918 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:21.647991 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:21.648288 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 03:09:21.648374 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":279,"threads_started":1}
I20260928 03:09:21.650223 24865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:21.636022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:38817 (local address 127.6.192.1:50489)
0928 03:09:21.637371 (+ 1349us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:21.637444 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:21.639474 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:21.640263 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:21.640277 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:21.640425 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:21.641405 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:21.641447 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:21.642671 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:21.642681 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:21.645154 (+ 2473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:21.645175 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:21.646107 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:21.646118 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:21.646521 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:21.647371 (+ 850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:09:21.647378 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:21.647472 (+ 94us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:09:21.647600 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:21.648026 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:21.648041 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:21.648054 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 03:09:21.648263 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 03:09:21.648763 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":373,"threads_started":1}
I20260928 03:09:21.652161 24862 heartbeater.cc:347] Connected to a master server at 127.6.192.62:38817
I20260928 03:09:21.652586 24862 heartbeater.cc:464] Registering TS with master...
I20260928 03:09:21.653515 24862 heartbeater.cc:511] Master 127.6.192.62:38817 requested a full tablet report, sending...
I20260928 03:09:21.656255 24676 ts_manager.cc:194] Registered new tserver with Master: 29f4a10c98714ef8a5075114b6236cf4 (127.6.192.1:45411)
I20260928 03:09:21.657753 24676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.192.1:50489
W20260928 03:09:21.989400 24866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:21.989754 24866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:21.989924 24866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:21.997661 24866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:21.997819 24866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:21.998024 24866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:09:22.007555 24866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:38817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:22.008970 24866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:22.010876 24866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:22.025903 24871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:22.025996 24874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:22.026163 24872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:22.026954 24866 server_base.cc:1034] running on GCE node
I20260928 03:09:22.027758 24866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:22.028770 24866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:22.029999 24866 hybrid_clock.cc:648] HybridClock initialized: now 1790564962029937 us; error 54 us; skew 500 ppm
I20260928 03:09:22.030306 24866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:09:22.033205 24866 webserver.cc:460] Webserver started at http://127.6.192.2:37779/ using document root <none> and password file <none>
I20260928 03:09:22.033917 24866 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:22.034008 24866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:22.034289 24866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:22.037312 24866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "6826e25aacd64d829826ce5fe188bafa"
format_stamp: "Formatted at 2026-09-28 03:09:22 on dist-test-slave-fjww"
server_key: "2dc83109354be3868eb3fc0e041eb92c"
server_key_iv: "1fd2c85bb6071aed72f87056edaaa810"
server_key_version: "encryptionkey@0"
I20260928 03:09:22.038205 24866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "6826e25aacd64d829826ce5fe188bafa"
format_stamp: "Formatted at 2026-09-28 03:09:22 on dist-test-slave-fjww"
server_key: "2dc83109354be3868eb3fc0e041eb92c"
server_key_iv: "1fd2c85bb6071aed72f87056edaaa810"
server_key_version: "encryptionkey@0"
I20260928 03:09:22.044075 24866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 03:09:22.047950 24880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:22.049548 24866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 03:09:22.049752 24866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "6826e25aacd64d829826ce5fe188bafa"
format_stamp: "Formatted at 2026-09-28 03:09:22 on dist-test-slave-fjww"
server_key: "2dc83109354be3868eb3fc0e041eb92c"
server_key_iv: "1fd2c85bb6071aed72f87056edaaa810"
server_key_version: "encryptionkey@0"
I20260928 03:09:22.049993 24866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:22.069608 24866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:22.070564 24866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:22.070879 24866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:22.072199 24866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:22.073977 24866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:22.074115 24866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:22.074302 24866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:22.074409 24866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:22.110844 24866 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:45661
I20260928 03:09:22.110885 24992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:45661 every 8 connection(s)
I20260928 03:09:22.112340 24866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:09:22.121222 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 24866
I20260928 03:09:22.121440 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:09:22.122077 6912 external_mini_cluster.cc:1428] Setting key 07e21b231f61c9aca499d6242e349306
I20260928 03:09:22.125969 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:38817
--builtin_ntp_servers=127.6.192.20:38565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 03:09:22.127331 24864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.115230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:40707 (local address 127.6.192.62:38817)
0928 03:09:22.115426 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:22.115431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:22.117147 (+ 1716us) server_negotiation.cc:407] Connection header received
0928 03:09:22.119054 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:22.119059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:22.119140 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:22.119240 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:22.120544 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.121539 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.123739 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.124626 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.124780 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.126506 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:22.126511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:22.126515 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:22.126543 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:22.126581 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:22.126594 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:22.126668 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:22.126964 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 03:09:22.127053 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 03:09:22.128937 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.115886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:38817 (local address 127.6.192.2:40707)
0928 03:09:22.116953 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:22.116997 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:22.118735 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:22.119604 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:22.119613 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:22.119717 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:22.120334 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:22.120352 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.121722 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.121729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:22.123582 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:22.123598 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.124910 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.124919 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:22.125247 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.126090 (+ 843us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:09:22.126095 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:22.126160 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:09:22.126278 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:22.126700 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:22.126715 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:22.126726 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 03:09:22.126932 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 03:09:22.127417 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":381,"threads_started":1}
I20260928 03:09:22.130674 24993 heartbeater.cc:347] Connected to a master server at 127.6.192.62:38817
I20260928 03:09:22.131045 24993 heartbeater.cc:464] Registering TS with master...
I20260928 03:09:22.132128 24993 heartbeater.cc:511] Master 127.6.192.62:38817 requested a full tablet report, sending...
I20260928 03:09:22.133971 24676 ts_manager.cc:194] Registered new tserver with Master: 6826e25aacd64d829826ce5fe188bafa (127.6.192.2:45661)
I20260928 03:09:22.135571 24676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.192.2:40707
W20260928 03:09:22.474546 24996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:22.474895 24996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:22.474990 24996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:22.482473 24996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:22.482602 24996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:22.482810 24996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:09:22.492364 24996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:38817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:22.493749 24996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:22.495710 24996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:22.510000 25004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:22.510172 25001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:22.510102 25002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:22.511170 24996 server_base.cc:1034] running on GCE node
I20260928 03:09:22.511924 24996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:22.513010 24996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:22.514276 24996 hybrid_clock.cc:648] HybridClock initialized: now 1790564962514219 us; error 80 us; skew 500 ppm
I20260928 03:09:22.514679 24996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:09:22.517580 24996 webserver.cc:460] Webserver started at http://127.6.192.3:40139/ using document root <none> and password file <none>
I20260928 03:09:22.518350 24996 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:22.518487 24996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:22.518846 24996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:22.522199 24996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "056de269739844be8f4ddbc1e628201f"
format_stamp: "Formatted at 2026-09-28 03:09:22 on dist-test-slave-fjww"
server_key: "8332867665960bca6f4c1f9be5edc601"
server_key_iv: "d4aa4e97e296ab2f7b4dfb3eeb20352e"
server_key_version: "encryptionkey@0"
I20260928 03:09:22.523206 24996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "056de269739844be8f4ddbc1e628201f"
format_stamp: "Formatted at 2026-09-28 03:09:22 on dist-test-slave-fjww"
server_key: "8332867665960bca6f4c1f9be5edc601"
server_key_iv: "d4aa4e97e296ab2f7b4dfb3eeb20352e"
server_key_version: "encryptionkey@0"
I20260928 03:09:22.528877 24996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 03:09:22.533106 25010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:22.534933 24996 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:09:22.535172 24996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "056de269739844be8f4ddbc1e628201f"
format_stamp: "Formatted at 2026-09-28 03:09:22 on dist-test-slave-fjww"
server_key: "8332867665960bca6f4c1f9be5edc601"
server_key_iv: "d4aa4e97e296ab2f7b4dfb3eeb20352e"
server_key_version: "encryptionkey@0"
I20260928 03:09:22.535451 24996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:22.555619 24996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:22.556643 24996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:22.556972 24996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:22.558290 24996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:22.560137 24996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:22.560240 24996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:22.560323 24996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:22.560386 24996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:22.597368 24996 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:43029
I20260928 03:09:22.597424 25122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:43029 every 8 connection(s)
I20260928 03:09:22.598862 24996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:09:22.606011 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 24996
I20260928 03:09:22.606207 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:09:22.606774 6912 external_mini_cluster.cc:1428] Setting key a918ac5c4fbc21e0456635b1cfc7ec2b
I20260928 03:09:22.612640 24864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.601698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:56671 (local address 127.6.192.62:38817)
0928 03:09:22.601918 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:22.601925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:22.603373 (+ 1448us) server_negotiation.cc:407] Connection header received
0928 03:09:22.605295 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:22.605299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:22.605383 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:22.605478 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:22.607049 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.607984 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.609661 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.610083 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.610216 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.611679 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:22.611683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:22.611687 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:22.611705 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:22.611738 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:22.611750 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:22.611853 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:22.612158 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 03:09:22.612259 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 03:09:22.613693 25125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.602274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:38817 (local address 127.6.192.3:56671)
0928 03:09:22.603202 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:22.603244 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:22.605019 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:22.605801 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:22.605816 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:22.605943 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:22.606819 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:22.606848 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.608148 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.608159 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:22.609468 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:22.609480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.610320 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.610329 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:22.610563 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.611230 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:09:22.611233 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:22.611276 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:09:22.611412 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:22.611931 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:22.611939 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:22.611946 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 03:09:22.612133 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 03:09:22.612568 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":307,"threads_started":1}
I20260928 03:09:22.615429 25123 heartbeater.cc:347] Connected to a master server at 127.6.192.62:38817
I20260928 03:09:22.616112 25123 heartbeater.cc:464] Registering TS with master...
I20260928 03:09:22.616932 25123 heartbeater.cc:511] Master 127.6.192.62:38817 requested a full tablet report, sending...
I20260928 03:09:22.618456 24676 ts_manager.cc:194] Registered new tserver with Master: 056de269739844be8f4ddbc1e628201f (127.6.192.3:43029)
I20260928 03:09:22.619320 24676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.192.3:56671
I20260928 03:09:22.622232 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:09:22.633364 24864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.626705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.6.192.62:38817)
0928 03:09:22.626889 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:22.626894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:22.627282 (+ 388us) server_negotiation.cc:407] Connection header received
0928 03:09:22.627552 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:22.627557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:22.627646 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:22.627744 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:22.628239 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.629012 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.630621 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.631289 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.631429 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.632581 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:22.632586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:22.632591 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:22.632616 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:22.632656 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:22.632671 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:22.632793 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:22.632905 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 03:09:22.633014 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 03:09:22.637712 24676 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:09:22.645123 24676 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:09:22.661397 24862 heartbeater.cc:503] Master 127.6.192.62:38817 was elected leader, sending a full tablet report...
I20260928 03:09:22.680716 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.658527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:43029 (local address 127.0.0.1:37302)
0928 03:09:22.669622 (+ 11095us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:22.669652 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:22.669862 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:22.671261 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:22.671274 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:22.671323 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:22.671889 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:22.671903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.674067 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.674072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:22.677205 (+ 3133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:22.677219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.679681 (+ 2462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.679690 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:22.679928 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.679993 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 03:09:22.680113 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 03:09:22.680320 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10850,"thread_start_us":13064,"threads_started":1}
I20260928 03:09:22.681521 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.659790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.6.192.3:43029)
0928 03:09:22.666702 (+ 6912us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:22.666717 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:22.669836 (+ 3119us) server_negotiation.cc:407] Connection header received
0928 03:09:22.670106 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:22.670120 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:22.670526 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:22.670818 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:22.672160 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.673849 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.677455 (+ 3606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.679018 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.679155 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.679549 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:22.680884 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0928 03:09:22.681139 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6646,"thread_start_us":6419,"threads_started":1}
I20260928 03:09:22.682181 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.660185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57104 (local address 127.6.192.1:45411)
0928 03:09:22.672174 (+ 11989us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:22.672184 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:22.673091 (+ 907us) server_negotiation.cc:407] Connection header received
0928 03:09:22.673375 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:22.673396 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:22.673659 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:22.673828 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:22.674838 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.676441 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.678557 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.680176 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.680419 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.680724 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:22.681766 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0928 03:09:22.681918 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11746,"thread_start_us":11833,"threads_started":1}
I20260928 03:09:22.682706 25136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.659317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:45411 (local address 127.0.0.1:57104)
0928 03:09:22.672979 (+ 13662us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:22.672999 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:22.673200 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:22.674086 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:22.674090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:22.674115 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:22.674640 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:22.674653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.676589 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.676594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:22.678359 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:22.678372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.680473 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.680481 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:22.680686 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.680730 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 03:09:22.681263 (+ 533us) client_negotiation.cc:240] Negotiation successful
0928 03:09:22.681426 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13526,"mutex_wait_us":70,"thread_start_us":9850,"threads_started":1}
I20260928 03:09:22.682753 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.658334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:45661 (local address 127.0.0.1:39222)
0928 03:09:22.659265 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:22.659317 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:22.659620 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:22.675195 (+ 15575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:22.675200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:22.675226 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:22.675801 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:22.675815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.677297 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.677302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:22.679497 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:22.679518 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.681342 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.681349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:22.681536 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.681607 (+ 71us) client_negotiation.cc:769] Sending connection context
0928 03:09:22.681746 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 03:09:22.681885 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":105,"thread_start_us":429,"threads_started":1}
I20260928 03:09:22.683182 25137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.658647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.6.192.2:45661)
0928 03:09:22.674561 (+ 15914us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:22.674572 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:22.674606 (+ 34us) server_negotiation.cc:407] Connection header received
0928 03:09:22.674714 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:22.674725 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:22.674978 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:22.675188 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:22.676007 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.677143 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.679735 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.681168 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.681306 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.681543 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:22.682027 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 03:09:22.682188 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15733,"thread_start_us":15605,"threads_started":1}
I20260928 03:09:22.686167 25058 tablet_service.cc:1467] Processing CreateTablet for tablet 8d6f7adad3b244aaaf31e12d491ff898 (DEFAULT_TABLE table=test-table [id=d1adafbd8c61434bbb18936ad707f491]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:09:22.686527 24797 tablet_service.cc:1467] Processing CreateTablet for tablet 8d6f7adad3b244aaaf31e12d491ff898 (DEFAULT_TABLE table=test-table [id=d1adafbd8c61434bbb18936ad707f491]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:09:22.687071 24928 tablet_service.cc:1467] Processing CreateTablet for tablet 8d6f7adad3b244aaaf31e12d491ff898 (DEFAULT_TABLE table=test-table [id=d1adafbd8c61434bbb18936ad707f491]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:09:22.687875 24797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d6f7adad3b244aaaf31e12d491ff898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:22.687947 25058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d6f7adad3b244aaaf31e12d491ff898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:22.688591 24928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d6f7adad3b244aaaf31e12d491ff898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:22.700345 25141 tablet_bootstrap.cc:492] T 8d6f7adad3b244aaaf31e12d491ff898 P 29f4a10c98714ef8a5075114b6236cf4: Bootstrap starting.
I20260928 03:09:22.704764 25143 tablet_bootstrap.cc:492] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f: Bootstrap starting.
I20260928 03:09:22.706246 25142 tablet_bootstrap.cc:492] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa: Bootstrap starting.
I20260928 03:09:22.706971 25141 tablet_bootstrap.cc:654] T 8d6f7adad3b244aaaf31e12d491ff898 P 29f4a10c98714ef8a5075114b6236cf4: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:22.708433 25141 log.cc:824] T 8d6f7adad3b244aaaf31e12d491ff898 P 29f4a10c98714ef8a5075114b6236cf4: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:22.708918 25143 tablet_bootstrap.cc:654] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:22.709905 25142 tablet_bootstrap.cc:654] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:22.710600 25143 log.cc:824] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:22.711087 25142 log.cc:824] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:22.714756 25143 tablet_bootstrap.cc:492] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f: No bootstrap required, opened a new log
I20260928 03:09:22.714789 25141 tablet_bootstrap.cc:492] T 8d6f7adad3b244aaaf31e12d491ff898 P 29f4a10c98714ef8a5075114b6236cf4: No bootstrap required, opened a new log
I20260928 03:09:22.715121 25143 ts_tablet_manager.cc:1397] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260928 03:09:22.715188 25141 ts_tablet_manager.cc:1397] T 8d6f7adad3b244aaaf31e12d491ff898 P 29f4a10c98714ef8a5075114b6236cf4: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260928 03:09:22.716068 25142 tablet_bootstrap.cc:492] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa: No bootstrap required, opened a new log
I20260928 03:09:22.716365 25142 ts_tablet_manager.cc:1397] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 03:09:22.723706 25143 raft_consensus.cc:348] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6826e25aacd64d829826ce5fe188bafa" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45661 } } peers { permanent_uuid: "056de269739844be8f4ddbc1e628201f" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43029 } } peers { permanent_uuid: "29f4a10c98714ef8a5075114b6236cf4" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45411 } }
I20260928 03:09:22.724332 25143 raft_consensus.cc:374] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:22.724462 25143 raft_consensus.cc:729] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 056de269739844be8f4ddbc1e628201f, State: Initialized, Role: FOLLOWER
I20260928 03:09:22.725381 25143 consensus_queue.cc:260] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6826e25aacd64d829826ce5fe188bafa" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45661 } } peers { permanent_uuid: "056de269739844be8f4ddbc1e628201f" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43029 } } peers { permanent_uuid: "29f4a10c98714ef8a5075114b6236cf4" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45411 } }
I20260928 03:09:22.726727 25141 raft_consensus.cc:348] T 8d6f7adad3b244aaaf31e12d491ff898 P 29f4a10c98714ef8a5075114b6236cf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6826e25aacd64d829826ce5fe188bafa" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45661 } } peers { permanent_uuid: "056de269739844be8f4ddbc1e628201f" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43029 } } peers { permanent_uuid: "29f4a10c98714ef8a5075114b6236cf4" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45411 } }
I20260928 03:09:22.727303 25141 raft_consensus.cc:374] T 8d6f7adad3b244aaaf31e12d491ff898 P 29f4a10c98714ef8a5075114b6236cf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:22.727409 25141 raft_consensus.cc:729] T 8d6f7adad3b244aaaf31e12d491ff898 P 29f4a10c98714ef8a5075114b6236cf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f4a10c98714ef8a5075114b6236cf4, State: Initialized, Role: FOLLOWER
I20260928 03:09:22.727651 25142 raft_consensus.cc:348] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6826e25aacd64d829826ce5fe188bafa" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45661 } } peers { permanent_uuid: "056de269739844be8f4ddbc1e628201f" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43029 } } peers { permanent_uuid: "29f4a10c98714ef8a5075114b6236cf4" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45411 } }
I20260928 03:09:22.728147 25142 raft_consensus.cc:374] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:22.728250 25142 raft_consensus.cc:729] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6826e25aacd64d829826ce5fe188bafa, State: Initialized, Role: FOLLOWER
I20260928 03:09:22.728310 25141 consensus_queue.cc:260] T 8d6f7adad3b244aaaf31e12d491ff898 P 29f4a10c98714ef8a5075114b6236cf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6826e25aacd64d829826ce5fe188bafa" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45661 } } peers { permanent_uuid: "056de269739844be8f4ddbc1e628201f" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43029 } } peers { permanent_uuid: "29f4a10c98714ef8a5075114b6236cf4" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45411 } }
I20260928 03:09:22.729357 25123 heartbeater.cc:503] Master 127.6.192.62:38817 was elected leader, sending a full tablet report...
I20260928 03:09:22.729101 25142 consensus_queue.cc:260] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6826e25aacd64d829826ce5fe188bafa" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45661 } } peers { permanent_uuid: "056de269739844be8f4ddbc1e628201f" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43029 } } peers { permanent_uuid: "29f4a10c98714ef8a5075114b6236cf4" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45411 } }
I20260928 03:09:22.730611 24993 heartbeater.cc:503] Master 127.6.192.62:38817 was elected leader, sending a full tablet report...
I20260928 03:09:22.731772 25142 ts_tablet_manager.cc:1428] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260928 03:09:22.733752 25141 ts_tablet_manager.cc:1428] T 8d6f7adad3b244aaaf31e12d491ff898 P 29f4a10c98714ef8a5075114b6236cf4: Time spent starting tablet: real 0.018s user 0.008s sys 0.011s
I20260928 03:09:22.735078 25143 ts_tablet_manager.cc:1428] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f: Time spent starting tablet: real 0.020s user 0.012s sys 0.004s
I20260928 03:09:22.766521 25149 raft_consensus.cc:482] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:09:22.766937 25149 raft_consensus.cc:504] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6826e25aacd64d829826ce5fe188bafa" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45661 } } peers { permanent_uuid: "056de269739844be8f4ddbc1e628201f" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43029 } } peers { permanent_uuid: "29f4a10c98714ef8a5075114b6236cf4" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45411 } }
I20260928 03:09:22.768636 25147 raft_consensus.cc:482] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:09:22.768991 25149 leader_election.cc:288] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 056de269739844be8f4ddbc1e628201f (127.6.192.3:43029), 29f4a10c98714ef8a5075114b6236cf4 (127.6.192.1:45411)
I20260928 03:09:22.768958 25147 raft_consensus.cc:504] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6826e25aacd64d829826ce5fe188bafa" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45661 } } peers { permanent_uuid: "056de269739844be8f4ddbc1e628201f" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43029 } } peers { permanent_uuid: "29f4a10c98714ef8a5075114b6236cf4" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45411 } }
I20260928 03:09:22.771095 25147 leader_election.cc:288] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6826e25aacd64d829826ce5fe188bafa (127.6.192.2:45661), 29f4a10c98714ef8a5075114b6236cf4 (127.6.192.1:45411)
I20260928 03:09:22.782001 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.770221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:44253 (local address 127.6.192.1:45411)
0928 03:09:22.770447 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:22.770453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:22.771423 (+ 970us) server_negotiation.cc:407] Connection header received
0928 03:09:22.772410 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:22.772420 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:22.772542 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:22.772617 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:22.773709 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.775110 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.778647 (+ 3537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.780270 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.780435 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.780597 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:22.781404 (+ 807us) server_negotiation.cc:299] Negotiation successful
0928 03:09:22.781608 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 03:09:22.782032 25125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.771396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:45661 (local address 127.6.192.3:39867)
0928 03:09:22.771703 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:22.771722 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:22.771842 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:22.773114 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:22.773122 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:22.773158 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:22.773427 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:22.773442 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.775518 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.775527 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:22.778386 (+ 2859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:22.778410 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.780865 (+ 2455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.780876 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:22.781175 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.781234 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 03:09:22.781358 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 03:09:22.781585 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260928 03:09:22.783038 25137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.772484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:39867 (local address 127.6.192.2:45661)
0928 03:09:22.772636 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:22.772641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:22.772658 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:09:22.772708 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:22.772712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:22.772785 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:22.772895 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:22.773643 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.775183 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.778650 (+ 3467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.780425 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.780594 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.780736 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:22.782438 (+ 1702us) server_negotiation.cc:299] Negotiation successful
0928 03:09:22.782642 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 03:09:22.783175 25150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.769442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:45411 (local address 127.6.192.2:44253)
0928 03:09:22.771217 (+ 1775us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:22.771301 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:22.771521 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:22.772890 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:22.772898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:22.773010 (+ 112us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:22.773488 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:22.773504 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.775377 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.775382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:22.778330 (+ 2948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:22.778355 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.780828 (+ 2473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.780838 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:22.781130 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.781186 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 03:09:22.782323 (+ 1137us) client_negotiation.cc:240] Negotiation successful
0928 03:09:22.782537 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1652,"spinlock_wait_cycles":33024,"thread_start_us":1120,"threads_started":1}
I20260928 03:09:22.784252 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6f7adad3b244aaaf31e12d491ff898" candidate_uuid: "056de269739844be8f4ddbc1e628201f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6826e25aacd64d829826ce5fe188bafa" is_pre_election: true
I20260928 03:09:22.784780 24948 raft_consensus.cc:2454] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 056de269739844be8f4ddbc1e628201f in term 0.
I20260928 03:09:22.785718 25014 leader_election.cc:302] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 056de269739844be8f4ddbc1e628201f, 6826e25aacd64d829826ce5fe188bafa; no voters:
I20260928 03:09:22.786218 25147 raft_consensus.cc:2781] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:09:22.786377 25147 raft_consensus.cc:482] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:09:22.786497 25147 raft_consensus.cc:3037] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:22.789852 24817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6f7adad3b244aaaf31e12d491ff898" candidate_uuid: "6826e25aacd64d829826ce5fe188bafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f4a10c98714ef8a5075114b6236cf4" is_pre_election: true
I20260928 03:09:22.790525 24817 raft_consensus.cc:2454] T 8d6f7adad3b244aaaf31e12d491ff898 P 29f4a10c98714ef8a5075114b6236cf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6826e25aacd64d829826ce5fe188bafa in term 0.
I20260928 03:09:22.790817 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.769536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:37771 (local address 127.6.192.3:43029)
0928 03:09:22.769782 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:22.769788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:22.770543 (+ 755us) server_negotiation.cc:407] Connection header received
0928 03:09:22.770911 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:22.770919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:22.770999 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:22.771065 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:22.784206 (+ 13141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.785365 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.788164 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.789631 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.789756 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.789853 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:22.790377 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 03:09:22.790531 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":78}
I20260928 03:09:22.790704 25147 raft_consensus.cc:504] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6826e25aacd64d829826ce5fe188bafa" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45661 } } peers { permanent_uuid: "056de269739844be8f4ddbc1e628201f" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43029 } } peers { permanent_uuid: "29f4a10c98714ef8a5075114b6236cf4" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45411 } }
I20260928 03:09:22.791257 24883 leader_election.cc:302] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f4a10c98714ef8a5075114b6236cf4, 6826e25aacd64d829826ce5fe188bafa; no voters:
I20260928 03:09:22.791554 25147 leader_election.cc:288] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [CANDIDATE]: Term 1 election: Requested vote from peers 6826e25aacd64d829826ce5fe188bafa (127.6.192.2:45661), 29f4a10c98714ef8a5075114b6236cf4 (127.6.192.1:45411)
I20260928 03:09:22.791640 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.769390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:43029 (local address 127.6.192.2:37771)
0928 03:09:22.770404 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:22.770435 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:22.770694 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:22.771741 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:22.771746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:22.771775 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:22.784005 (+ 12230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:22.784023 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.785571 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.785576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:22.787931 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:22.787951 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.789913 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.789920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:22.790129 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.790171 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 03:09:22.791070 (+ 899us) client_negotiation.cc:240] Negotiation successful
0928 03:09:22.791223 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"thread_start_us":2708,"threads_started":1}
I20260928 03:09:22.791822 25149 raft_consensus.cc:2781] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:09:22.791966 25149 raft_consensus.cc:482] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:09:22.792075 25149 raft_consensus.cc:3037] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:22.792619 25078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6f7adad3b244aaaf31e12d491ff898" candidate_uuid: "6826e25aacd64d829826ce5fe188bafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "056de269739844be8f4ddbc1e628201f" is_pre_election: true
I20260928 03:09:22.793180 25078 raft_consensus.cc:2379] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6826e25aacd64d829826ce5fe188bafa in current term 1: Already voted for candidate 056de269739844be8f4ddbc1e628201f in this term.
I20260928 03:09:22.793576 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6f7adad3b244aaaf31e12d491ff898" candidate_uuid: "056de269739844be8f4ddbc1e628201f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6826e25aacd64d829826ce5fe188bafa"
I20260928 03:09:22.794056 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.771633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:45115 (local address 127.6.192.1:45411)
0928 03:09:22.782282 (+ 10649us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:22.782287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:22.784106 (+ 1819us) server_negotiation.cc:407] Connection header received
0928 03:09:22.784469 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:22.784477 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:22.784571 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:22.784630 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:22.785735 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.787198 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.791649 (+ 4451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.792898 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.793033 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.793136 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:22.793612 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 03:09:22.793771 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10548,"thread_start_us":17109,"threads_started":1}
I20260928 03:09:22.794744 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.771711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:45411 (local address 127.6.192.3:45115)
0928 03:09:22.783983 (+ 12272us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:22.784006 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:22.784256 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:22.784871 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:22.784892 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:22.784925 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:22.785506 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:22.785527 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.787502 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.787511 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:22.791249 (+ 3738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:22.791297 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.793122 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.793129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:22.793395 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.793438 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 03:09:22.794278 (+ 840us) client_negotiation.cc:240] Negotiation successful
0928 03:09:22.794427 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12081,"mutex_wait_us":1,"thread_start_us":11810,"threads_started":1}
I20260928 03:09:22.795421 24817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6f7adad3b244aaaf31e12d491ff898" candidate_uuid: "056de269739844be8f4ddbc1e628201f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f4a10c98714ef8a5075114b6236cf4" is_pre_election: true
I20260928 03:09:22.795457 25149 raft_consensus.cc:504] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6826e25aacd64d829826ce5fe188bafa" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45661 } } peers { permanent_uuid: "056de269739844be8f4ddbc1e628201f" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43029 } } peers { permanent_uuid: "29f4a10c98714ef8a5075114b6236cf4" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45411 } }
I20260928 03:09:22.795707 24817 raft_consensus.cc:2454] T 8d6f7adad3b244aaaf31e12d491ff898 P 29f4a10c98714ef8a5075114b6236cf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 056de269739844be8f4ddbc1e628201f in term 0.
I20260928 03:09:22.795826 24816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6f7adad3b244aaaf31e12d491ff898" candidate_uuid: "056de269739844be8f4ddbc1e628201f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f4a10c98714ef8a5075114b6236cf4"
I20260928 03:09:22.796209 25149 leader_election.cc:288] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [CANDIDATE]: Term 1 election: Requested vote from peers 056de269739844be8f4ddbc1e628201f (127.6.192.3:43029), 29f4a10c98714ef8a5075114b6236cf4 (127.6.192.1:45411)
I20260928 03:09:22.796242 24816 raft_consensus.cc:3037] T 8d6f7adad3b244aaaf31e12d491ff898 P 29f4a10c98714ef8a5075114b6236cf4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:22.796540 24948 raft_consensus.cc:2379] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 056de269739844be8f4ddbc1e628201f in current term 1: Already voted for candidate 6826e25aacd64d829826ce5fe188bafa in this term.
I20260928 03:09:22.798189 25078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6f7adad3b244aaaf31e12d491ff898" candidate_uuid: "6826e25aacd64d829826ce5fe188bafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "056de269739844be8f4ddbc1e628201f"
I20260928 03:09:22.798452 25078 raft_consensus.cc:2379] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6826e25aacd64d829826ce5fe188bafa in current term 1: Already voted for candidate 056de269739844be8f4ddbc1e628201f in this term.
I20260928 03:09:22.799726 24816 raft_consensus.cc:2454] T 8d6f7adad3b244aaaf31e12d491ff898 P 29f4a10c98714ef8a5075114b6236cf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 056de269739844be8f4ddbc1e628201f in term 1.
I20260928 03:09:22.799736 24817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6f7adad3b244aaaf31e12d491ff898" candidate_uuid: "6826e25aacd64d829826ce5fe188bafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f4a10c98714ef8a5075114b6236cf4"
I20260928 03:09:22.800012 24817 raft_consensus.cc:2379] T 8d6f7adad3b244aaaf31e12d491ff898 P 29f4a10c98714ef8a5075114b6236cf4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6826e25aacd64d829826ce5fe188bafa in current term 1: Already voted for candidate 056de269739844be8f4ddbc1e628201f in this term.
I20260928 03:09:22.800609 25013 leader_election.cc:302] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 056de269739844be8f4ddbc1e628201f, 29f4a10c98714ef8a5075114b6236cf4; no voters: 6826e25aacd64d829826ce5fe188bafa
I20260928 03:09:22.800750 24883 leader_election.cc:302] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6826e25aacd64d829826ce5fe188bafa; no voters: 056de269739844be8f4ddbc1e628201f, 29f4a10c98714ef8a5075114b6236cf4
I20260928 03:09:22.801041 25147 raft_consensus.cc:2781] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:22.801126 25149 raft_consensus.cc:2726] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 03:09:22.801636 25147 raft_consensus.cc:686] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [term 1 LEADER]: Becoming Leader. State: Replica: 056de269739844be8f4ddbc1e628201f, State: Running, Role: LEADER
I20260928 03:09:22.802266 25147 consensus_queue.cc:237] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6826e25aacd64d829826ce5fe188bafa" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45661 } } peers { permanent_uuid: "056de269739844be8f4ddbc1e628201f" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43029 } } peers { permanent_uuid: "29f4a10c98714ef8a5075114b6236cf4" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45411 } }
I20260928 03:09:22.809201 24675 catalog_manager.cc:5495] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f reported cstate change: term changed from 0 to 1, leader changed from <none> to 056de269739844be8f4ddbc1e628201f (127.6.192.3). New cstate: current_term: 1 leader_uuid: "056de269739844be8f4ddbc1e628201f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6826e25aacd64d829826ce5fe188bafa" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "056de269739844be8f4ddbc1e628201f" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 43029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29f4a10c98714ef8a5075114b6236cf4" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45411 } health_report { overall_health: UNKNOWN } } }
I20260928 03:09:22.833047 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.827082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37310 (local address 127.6.192.3:43029)
0928 03:09:22.827291 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:22.827298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:22.827318 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:09:22.827630 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:22.827634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:22.827728 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:22.827900 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:09:22.828251 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.829039 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.830660 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.831101 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.831240 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.831876 (+ 636us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:09:22.832412 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:09:22.832579 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:22.832640 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 03:09:22.832726 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 03:09:22.841859 24817 raft_consensus.cc:1261] T 8d6f7adad3b244aaaf31e12d491ff898 P 29f4a10c98714ef8a5075114b6236cf4 [term 1 FOLLOWER]: Refusing update from remote peer 056de269739844be8f4ddbc1e628201f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:09:22.841997 24948 raft_consensus.cc:1261] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [term 1 FOLLOWER]: Refusing update from remote peer 056de269739844be8f4ddbc1e628201f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:09:22.842832 25154 consensus_queue.cc:1035] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f4a10c98714ef8a5075114b6236cf4" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 45411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:09:22.843227 25147 consensus_queue.cc:1035] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6826e25aacd64d829826ce5fe188bafa" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 03:09:22.850808 25124 tablet.cc:2369] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:09:22.860801 25159 mvcc.cc:205] Tried to move back new op lower bound from 7334154087789686784 to 7334154087643197440. Current Snapshot: MvccSnapshot[applied={T|T < 7334154087789686784}]
I20260928 03:09:22.872780 24676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:09:22.873113 24676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:09:22.877442 24676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ea479b044f44169afd6233496a55889: Sending DeleteTablet for 3 replicas of tablet 8d6f7adad3b244aaaf31e12d491ff898
I20260928 03:09:22.878777 24928 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d6f7adad3b244aaaf31e12d491ff898 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:09:22 UTC) from {username='slave'} at 127.0.0.1:39222
I20260928 03:09:22.878870 24797 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d6f7adad3b244aaaf31e12d491ff898 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:09:22 UTC) from {username='slave'} at 127.0.0.1:57104
I20260928 03:09:22.878957 25058 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d6f7adad3b244aaaf31e12d491ff898 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:09:22 UTC) from {username='slave'} at 127.0.0.1:37302
I20260928 03:09:22.881397 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 24735
I20260928 03:09:22.891163 25169 tablet_replica.cc:330] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f: stopping tablet replica
I20260928 03:09:22.891760 25169 raft_consensus.cc:2229] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:09:22.892724 25169 raft_consensus.cc:2258] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:09:22.898013 25168 tablet_replica.cc:330] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa: stopping tablet replica
I20260928 03:09:22.898713 25168 raft_consensus.cc:2229] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:09:22.899729 25168 raft_consensus.cc:2258] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:09:22.900665 25167 tablet_replica.cc:330] T 8d6f7adad3b244aaaf31e12d491ff898 P 29f4a10c98714ef8a5075114b6236cf4: stopping tablet replica
I20260928 03:09:22.901186 25167 raft_consensus.cc:2229] T 8d6f7adad3b244aaaf31e12d491ff898 P 29f4a10c98714ef8a5075114b6236cf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:09:22.901775 25167 raft_consensus.cc:2258] T 8d6f7adad3b244aaaf31e12d491ff898 P 29f4a10c98714ef8a5075114b6236cf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:09:22.902266 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:22.882693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57108 (local address 127.6.192.1:45411)
0928 03:09:22.882894 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:22.882900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:22.888135 (+ 5235us) server_negotiation.cc:407] Connection header received
0928 03:09:22.889238 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:22.889248 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:22.889409 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:22.889508 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:22.891154 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.892999 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.896638 (+ 3639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:22.897631 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:22.897823 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:22.900150 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:22.900169 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:22.900195 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:22.901138 (+ 943us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:22.901219 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:22.901247 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:22.901359 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:22.901698 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 03:09:22.901846 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 03:09:22.902297 25169 ts_tablet_manager.cc:1905] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:09:22.902462 25168 ts_tablet_manager.cc:1905] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:09:22.903836 24857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:22.910014 25169 ts_tablet_manager.cc:1918] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:09:22.910249 25169 log.cc:1196] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/8d6f7adad3b244aaaf31e12d491ff898
I20260928 03:09:22.910763 25169 ts_tablet_manager.cc:1939] T 8d6f7adad3b244aaaf31e12d491ff898 P 056de269739844be8f4ddbc1e628201f: Deleting consensus metadata
I20260928 03:09:22.912818 24660 catalog_manager.cc:4841] TS 056de269739844be8f4ddbc1e628201f (127.6.192.3:43029): tablet 8d6f7adad3b244aaaf31e12d491ff898 (table test-table [id=d1adafbd8c61434bbb18936ad707f491]) successfully deleted
I20260928 03:09:22.913961 25168 ts_tablet_manager.cc:1918] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:09:22.914139 25168 log.cc:1196] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/8d6f7adad3b244aaaf31e12d491ff898
I20260928 03:09:22.914674 25168 ts_tablet_manager.cc:1939] T 8d6f7adad3b244aaaf31e12d491ff898 P 6826e25aacd64d829826ce5fe188bafa: Deleting consensus metadata
I20260928 03:09:22.916685 24663 catalog_manager.cc:4841] TS 6826e25aacd64d829826ce5fe188bafa (127.6.192.2:45661): tablet 8d6f7adad3b244aaaf31e12d491ff898 (table test-table [id=d1adafbd8c61434bbb18936ad707f491]) successfully deleted
I20260928 03:09:23.000362 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 24735
I20260928 03:09:23.026808 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 24866
W20260928 03:09:23.026623 24662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.192.1:45411 (error 108)
I20260928 03:09:23.028949 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:23.028394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:45411 (local address 127.0.0.1:57122)
0928 03:09:23.028608 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:23.028754 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:45411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260928 03:09:23.029294 24662 catalog_manager.cc:4584] TS 29f4a10c98714ef8a5075114b6236cf4 (127.6.192.1:45411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d6f7adad3b244aaaf31e12d491ff898: Network error: Client connection negotiation failed: client connection to 127.6.192.1:45411: connect: Connection refused (error 111)
I20260928 03:09:23.036644 25137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:23.027756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.6.192.2:45661)
0928 03:09:23.027961 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:23.027968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:23.028605 (+ 637us) server_negotiation.cc:407] Connection header received
0928 03:09:23.028741 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:23.028746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:23.028837 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:23.028980 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:23.029501 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:23.030852 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:23.032630 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:23.033345 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:23.033493 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:23.034754 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:23.034768 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:23.034820 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:23.035731 (+ 911us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:23.035806 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:23.035834 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:23.035967 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:23.036097 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 03:09:23.036224 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 03:09:23.037484 24988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:23.083281 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:23.082625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:45411 (local address 127.0.0.1:57136)
0928 03:09:23.082877 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:23.083006 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:45411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 03:09:23.084614 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:23.084165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:45411 (local address 127.0.0.1:57146)
0928 03:09:23.084358 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:23.084451 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:45411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 03:09:23.140044 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 24866
I20260928 03:09:23.145301 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:23.144796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:45411 (local address 127.0.0.1:57154)
0928 03:09:23.145041 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:23.145139 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:45411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260928 03:09:23.146510 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:23.146126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:45411 (local address 127.0.0.1:57160)
0928 03:09:23.146295 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:23.146390 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:45411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 03:09:23.165632 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 24996
I20260928 03:09:23.174382 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:23.166548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.6.192.3:43029)
0928 03:09:23.166768 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:23.166775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:23.166808 (+ 33us) server_negotiation.cc:407] Connection header received
0928 03:09:23.166926 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:23.166938 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:23.167114 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:23.167262 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:23.167718 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:23.169011 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:23.170517 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:23.171358 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:23.171575 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:23.172838 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:23.172851 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:23.172871 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:23.173552 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:23.173606 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:23.173627 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:23.173773 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:23.173927 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 03:09:23.174039 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 03:09:23.175194 25118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:23.227520 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:23.226885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:45411 (local address 127.0.0.1:57174)
0928 03:09:23.227136 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:23.227241 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:45411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260928 03:09:23.228767 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:23.228341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:45411 (local address 127.0.0.1:57186)
0928 03:09:23.228554 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:23.228644 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:45411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260928 03:09:23.298862 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 24996
I20260928 03:09:23.319160 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 24645
I20260928 03:09:23.320478 24706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:23.434378 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:23.433776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:45411 (local address 127.0.0.1:57190)
0928 03:09:23.433986 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:23.434129 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:45411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 03:09:23.434378 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 24645
2026-09-28T03:09:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2940 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:09:23 dist-test-slave-fjww krb5kdc[25178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:09:23 dist-test-slave-fjww krb5kdc[25178](info): set up 2 sockets
Sep 28 03:09:23 dist-test-slave-fjww krb5kdc[25178](info): commencing operation
krb5kdc: starting...
W20260928 03:09:24.583993 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:09:24 dist-test-slave-fjww krb5kdc[25178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:09:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:09:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:09:24.876022 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:41207
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:38441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:41207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:25.228155 25191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:25.228538 25191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:25.228703 25191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:25.236603 25191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:09:25.236788 25191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:25.236912 25191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:25.237025 25191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:09:25.237104 25191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:09:25.247821 25191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:41207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:41207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:25.249354 25191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:25.250751 25191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:25.264062 25196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:25.264053 25197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:25.263969 25199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:25.264842 25191 server_base.cc:1034] running on GCE node
I20260928 03:09:25.265726 25191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:25.267397 25191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:25.268677 25191 hybrid_clock.cc:648] HybridClock initialized: now 1790564965268622 us; error 46 us; skew 500 ppm
I20260928 03:09:25.269116 25191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:25 dist-test-slave-fjww krb5kdc[25178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564965, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:25.278856 25191 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:09:25.280839 25191 webserver.cc:460] Webserver started at http://127.6.192.62:37071/ using document root <none> and password file <none>
I20260928 03:09:25.281641 25191 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:25.281761 25191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:25.282055 25191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:25.285347 25191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "856785a6970c4768879d349de9fa559b"
format_stamp: "Formatted at 2026-09-28 03:09:25 on dist-test-slave-fjww"
server_key: "8c7458224846378b93c726b8a99e5e4f"
server_key_iv: "8faf16a15f82047411a22373807abf49"
server_key_version: "encryptionkey@0"
I20260928 03:09:25.286163 25191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "856785a6970c4768879d349de9fa559b"
format_stamp: "Formatted at 2026-09-28 03:09:25 on dist-test-slave-fjww"
server_key: "8c7458224846378b93c726b8a99e5e4f"
server_key_iv: "8faf16a15f82047411a22373807abf49"
server_key_version: "encryptionkey@0"
I20260928 03:09:25.292263 25191 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260928 03:09:25.296276 25206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:25.297816 25191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:09:25.298035 25191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "856785a6970c4768879d349de9fa559b"
format_stamp: "Formatted at 2026-09-28 03:09:25 on dist-test-slave-fjww"
server_key: "8c7458224846378b93c726b8a99e5e4f"
server_key_iv: "8faf16a15f82047411a22373807abf49"
server_key_version: "encryptionkey@0"
I20260928 03:09:25.298251 25191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:25.319813 25191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:25.325397 25191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:25.325711 25191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:25.346284 25191 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:41207
I20260928 03:09:25.346315 25257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:41207 every 8 connection(s)
I20260928 03:09:25.347853 25191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:09:25.352006 25258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:25.353081 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 25191
I20260928 03:09:25.353279 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:09:25.353849 6912 external_mini_cluster.cc:1428] Setting key a65e7208626c1da1b9ed0c9283b47465
I20260928 03:09:25.366170 25258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b: Bootstrap starting.
Sep 28 03:09:25 dist-test-slave-fjww krb5kdc[25178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:09:25.369609 25258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:25.370700 25258 log.cc:824] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:25.374207 25258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b: No bootstrap required, opened a new log
I20260928 03:09:25.379771 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:25.356115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46986 (local address 127.6.192.62:41207)
0928 03:09:25.356995 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:25.357008 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:25.357053 (+ 45us) server_negotiation.cc:407] Connection header received
0928 03:09:25.359236 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:25.359265 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:25.359877 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:25.360367 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:25.362022 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:25.363423 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:25.364935 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:25.365676 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:25.365923 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:25.370500 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:25.370510 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:25.370527 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:25.370572 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:25.374223 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:25.375340 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:25.375347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:25.375354 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:25.375668 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:25.376202 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:25.376206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:25.376208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:25.376983 (+ 775us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:25.377270 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:25.377898 (+ 628us) server_negotiation.cc:299] Negotiation successful
0928 03:09:25.378435 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":423,"threads_started":1}
I20260928 03:09:25.384444 25258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856785a6970c4768879d349de9fa559b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41207 } }
I20260928 03:09:25.384894 25258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:25.385012 25258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856785a6970c4768879d349de9fa559b, State: Initialized, Role: FOLLOWER
I20260928 03:09:25.385613 25258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856785a6970c4768879d349de9fa559b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41207 } }
I20260928 03:09:25.385877 25258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:09:25.386047 25258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:09:25.386235 25258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:25.389065 25258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856785a6970c4768879d349de9fa559b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41207 } }
I20260928 03:09:25.389657 25258 leader_election.cc:302] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 856785a6970c4768879d349de9fa559b; no voters:
I20260928 03:09:25.390337 25258 leader_election.cc:288] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:09:25.390516 25263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:25.391335 25263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b [term 1 LEADER]: Becoming Leader. State: Replica: 856785a6970c4768879d349de9fa559b, State: Running, Role: LEADER
I20260928 03:09:25.392027 25263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856785a6970c4768879d349de9fa559b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41207 } }
I20260928 03:09:25.392917 25258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:09:25.395180 25264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "856785a6970c4768879d349de9fa559b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856785a6970c4768879d349de9fa559b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41207 } } }
I20260928 03:09:25.395681 25264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:25.395828 25265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 856785a6970c4768879d349de9fa559b. Latest consensus state: current_term: 1 leader_uuid: "856785a6970c4768879d349de9fa559b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856785a6970c4768879d349de9fa559b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 41207 } } }
I20260928 03:09:25.396129 25265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:25.397833 25272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:09:25.403244 25272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:09:25.412923 25272 catalog_manager.cc:1347] Generated new cluster ID: 3b1f7668823646aabc4a032240e4493b
I20260928 03:09:25.413074 25272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:09:25.454808 25272 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:09:25.456255 25272 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:09:25.464789 25272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b: Generated new TSK 0
I20260928 03:09:25.465799 25272 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
I20260928 03:09:25.548539 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41207
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:38441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:25.898985 25284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:25.899350 25284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:25.899544 25284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:25.907725 25284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:25.907894 25284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:25.908079 25284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:09:25.918483 25284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:25.920020 25284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:25.921583 25284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:25.935935 25289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:25.935961 25292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:25.935935 25290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:25.936959 25284 server_base.cc:1034] running on GCE node
I20260928 03:09:25.937597 25284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:25.938627 25284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:25.939833 25284 hybrid_clock.cc:648] HybridClock initialized: now 1790564965939794 us; error 100 us; skew 500 ppm
I20260928 03:09:25.940212 25284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:25 dist-test-slave-fjww krb5kdc[25178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564965, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:25.943539 25284 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:09:25.945992 25284 webserver.cc:460] Webserver started at http://127.6.192.1:42529/ using document root <none> and password file <none>
I20260928 03:09:25.947171 25284 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:25.947336 25284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:25.947791 25284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:25.952288 25284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "8d47b979fa98482db49690c94c1db976"
format_stamp: "Formatted at 2026-09-28 03:09:25 on dist-test-slave-fjww"
server_key: "7a69425fbbad8ba33c608bfdcc9f649b"
server_key_iv: "977756ef9e744efea7f2201ac5509f07"
server_key_version: "encryptionkey@0"
I20260928 03:09:25.953498 25284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "8d47b979fa98482db49690c94c1db976"
format_stamp: "Formatted at 2026-09-28 03:09:25 on dist-test-slave-fjww"
server_key: "7a69425fbbad8ba33c608bfdcc9f649b"
server_key_iv: "977756ef9e744efea7f2201ac5509f07"
server_key_version: "encryptionkey@0"
I20260928 03:09:25.959431 25284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 03:09:25.963380 25299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:25.965075 25284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:09:25.965293 25284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "8d47b979fa98482db49690c94c1db976"
format_stamp: "Formatted at 2026-09-28 03:09:25 on dist-test-slave-fjww"
server_key: "7a69425fbbad8ba33c608bfdcc9f649b"
server_key_iv: "977756ef9e744efea7f2201ac5509f07"
server_key_version: "encryptionkey@0"
I20260928 03:09:25.965529 25284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:25.999651 25284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:26.005369 25284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:26.005704 25284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:26.006788 25284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:26.008443 25284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:26.008519 25284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:26.008657 25284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:26.008721 25284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:26.046226 25284 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:35399
I20260928 03:09:26.046324 25411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:35399 every 8 connection(s)
I20260928 03:09:26.047637 25284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:09:26.049492 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 25284
I20260928 03:09:26.049691 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:09:26.050171 6912 external_mini_cluster.cc:1428] Setting key 504368759187a189164aa1d7e6b54eb1
Sep 28 03:09:26 dist-test-slave-fjww krb5kdc[25178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564965, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:09:26.072362 25414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:26.050910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:49883 (local address 127.6.192.62:41207)
0928 03:09:26.051583 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:26.051589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:26.052696 (+ 1107us) server_negotiation.cc:407] Connection header received
0928 03:09:26.054604 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:26.054610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:26.054698 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:26.054835 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:26.056567 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:26.057328 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:26.059034 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:26.059543 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:26.059689 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:26.064026 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:26.064035 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:26.064043 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:26.064093 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:26.068023 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:26.069114 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:26.069124 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:26.069131 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:26.069324 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:26.069883 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:26.069891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:26.069897 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:26.070487 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:26.070692 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:26.071548 (+ 856us) server_negotiation.cc:299] Negotiation successful
0928 03:09:26.071856 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":356,"threads_started":1}
I20260928 03:09:26.073279 25415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:26.051683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41207 (local address 127.6.192.1:49883)
0928 03:09:26.052536 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:26.052578 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:26.054305 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:26.055101 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:26.055110 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:26.055756 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:26.056363 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:26.056382 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:26.057520 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:26.057526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:26.058842 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:26.058855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:26.059829 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:26.059836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:26.060088 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:26.060962 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:09:26.060987 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:26.063731 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:26.068234 (+ 4503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:26.068244 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:26.068260 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:26.068872 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:26.069497 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:26.069500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:26.069503 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:26.069750 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:26.070636 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:26.070643 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:26.071067 (+ 424us) client_negotiation.cc:769] Sending connection context
0928 03:09:26.071333 (+ 266us) client_negotiation.cc:240] Negotiation successful
0928 03:09:26.071911 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":438,"threads_started":1}
I20260928 03:09:26.075210 25412 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41207
I20260928 03:09:26.075707 25412 heartbeater.cc:464] Registering TS with master...
I20260928 03:09:26.076565 25412 heartbeater.cc:511] Master 127.6.192.62:41207 requested a full tablet report, sending...
I20260928 03:09:26.079054 25223 ts_manager.cc:194] Registered new tserver with Master: 8d47b979fa98482db49690c94c1db976 (127.6.192.1:35399)
WARNING: no policy specified for kudu/127.6.192.2@KRBTEST.COM; defaulting to no policy
I20260928 03:09:26.080889 25223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:49883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
I20260928 03:09:26.115669 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41207
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:38441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:26.473102 25418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:26.473517 25418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:26.473713 25418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:26.482366 25418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:26.482539 25418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:26.482735 25418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:09:26.493547 25418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:26.494938 25418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:26.496373 25418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:26.510885 25426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:26.510988 25424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:26.510934 25423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:26.512130 25418 server_base.cc:1034] running on GCE node
I20260928 03:09:26.512902 25418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:26.513978 25418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:26.515249 25418 hybrid_clock.cc:648] HybridClock initialized: now 1790564966515188 us; error 67 us; skew 500 ppm
I20260928 03:09:26.515709 25418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:26 dist-test-slave-fjww krb5kdc[25178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564966, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:26.519390 25418 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:09:26.521180 25418 webserver.cc:460] Webserver started at http://127.6.192.2:45605/ using document root <none> and password file <none>
I20260928 03:09:26.521977 25418 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:26.522140 25418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:26.522495 25418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:26.525934 25418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "736d050cbd114d138b002eddcbaf0992"
format_stamp: "Formatted at 2026-09-28 03:09:26 on dist-test-slave-fjww"
server_key: "678258678496c984948097f70af73eca"
server_key_iv: "2a580e18f127699e223fd57ed98a2db5"
server_key_version: "encryptionkey@0"
I20260928 03:09:26.526840 25418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "736d050cbd114d138b002eddcbaf0992"
format_stamp: "Formatted at 2026-09-28 03:09:26 on dist-test-slave-fjww"
server_key: "678258678496c984948097f70af73eca"
server_key_iv: "2a580e18f127699e223fd57ed98a2db5"
server_key_version: "encryptionkey@0"
I20260928 03:09:26.533180 25418 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260928 03:09:26.537482 25433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:26.539080 25418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 03:09:26.539290 25418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "736d050cbd114d138b002eddcbaf0992"
format_stamp: "Formatted at 2026-09-28 03:09:26 on dist-test-slave-fjww"
server_key: "678258678496c984948097f70af73eca"
server_key_iv: "2a580e18f127699e223fd57ed98a2db5"
server_key_version: "encryptionkey@0"
I20260928 03:09:26.539642 25418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:26.563509 25418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:26.568828 25418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:26.569180 25418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:26.570429 25418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:26.572207 25418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:26.572332 25418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:26.572510 25418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:26.572588 25418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:26.610584 25418 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:42333
I20260928 03:09:26.610620 25545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:42333 every 8 connection(s)
I20260928 03:09:26.612104 25418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:09:26.613651 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 25418
I20260928 03:09:26.613978 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:09:26.614578 6912 external_mini_cluster.cc:1428] Setting key 4da8724daebce3aebeaabddd20dd14e0
Sep 28 03:09:26 dist-test-slave-fjww krb5kdc[25178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564966, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:09:26.636540 25548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:26.615256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:38841 (local address 127.6.192.62:41207)
0928 03:09:26.615805 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:26.615811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:26.617007 (+ 1196us) server_negotiation.cc:407] Connection header received
0928 03:09:26.618936 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:26.618942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:26.619042 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:26.619164 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:26.620967 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:26.621722 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:26.623478 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:26.623998 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:26.624157 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:26.628468 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:26.628478 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:26.628486 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:26.628537 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:26.632360 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:26.633377 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:26.633385 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:26.633392 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:26.633565 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:26.634160 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:26.634167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:26.634173 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:26.634571 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:26.634737 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:26.635727 (+ 990us) server_negotiation.cc:299] Negotiation successful
0928 03:09:26.636040 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":277,"threads_started":1}
I20260928 03:09:26.637329 25549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:26.615932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41207 (local address 127.6.192.2:38841)
0928 03:09:26.616835 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:26.616893 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:26.618643 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:26.619512 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:26.619524 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:26.620100 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:26.620767 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:26.620795 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:26.621894 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:26.621900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:26.623268 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:26.623283 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:26.624307 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:26.624314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:26.624558 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:26.625355 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:09:26.625386 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:26.628156 (+ 2770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:26.632574 (+ 4418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:26.632582 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:26.632598 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:26.633203 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:26.633694 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:26.633697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:26.633700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:26.634023 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:26.634814 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:26.634821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:26.635215 (+ 394us) client_negotiation.cc:769] Sending connection context
0928 03:09:26.635585 (+ 370us) client_negotiation.cc:240] Negotiation successful
0928 03:09:26.636059 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":322,"threads_started":1}
I20260928 03:09:26.639168 25546 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41207
I20260928 03:09:26.639590 25546 heartbeater.cc:464] Registering TS with master...
I20260928 03:09:26.640424 25546 heartbeater.cc:511] Master 127.6.192.62:41207 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.192.3@KRBTEST.COM; defaulting to no policy
I20260928 03:09:26.642105 25223 ts_manager.cc:194] Registered new tserver with Master: 736d050cbd114d138b002eddcbaf0992 (127.6.192.2:42333)
I20260928 03:09:26.643056 25223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:38841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
I20260928 03:09:26.679229 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41207
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:38441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:27.027511 25552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:27.027828 25552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:27.027937 25552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:27.035691 25552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:27.035810 25552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:27.035934 25552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:09:27.046626 25552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:41207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:27.048007 25552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:27.049355 25552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:27.063742 25557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:27.063730 25560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:27.063766 25558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:27.064689 25552 server_base.cc:1034] running on GCE node
I20260928 03:09:27.065447 25552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:27.066512 25552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:27.067747 25552 hybrid_clock.cc:648] HybridClock initialized: now 1790564967067690 us; error 62 us; skew 500 ppm
I20260928 03:09:27.068120 25552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:27 dist-test-slave-fjww krb5kdc[25178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564967, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:27.071759 25552 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:09:27.073421 25552 webserver.cc:460] Webserver started at http://127.6.192.3:35519/ using document root <none> and password file <none>
I20260928 03:09:27.074162 25552 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:27.074313 25552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:27.074640 25552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:27.077829 25552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "dec87d0d43624986a1a79ac24f62220c"
format_stamp: "Formatted at 2026-09-28 03:09:27 on dist-test-slave-fjww"
server_key: "e4e9fd5f1a4856edeb1f0fdcc5078ad4"
server_key_iv: "28f6cd46201048adf6a6ba3def2ff302"
server_key_version: "encryptionkey@0"
I20260928 03:09:27.078703 25552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "dec87d0d43624986a1a79ac24f62220c"
format_stamp: "Formatted at 2026-09-28 03:09:27 on dist-test-slave-fjww"
server_key: "e4e9fd5f1a4856edeb1f0fdcc5078ad4"
server_key_iv: "28f6cd46201048adf6a6ba3def2ff302"
server_key_version: "encryptionkey@0"
I20260928 03:09:27.084677 25552 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 03:09:27.085466 25412 heartbeater.cc:503] Master 127.6.192.62:41207 was elected leader, sending a full tablet report...
I20260928 03:09:27.089011 25567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:27.090607 25552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:09:27.090816 25552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "dec87d0d43624986a1a79ac24f62220c"
format_stamp: "Formatted at 2026-09-28 03:09:27 on dist-test-slave-fjww"
server_key: "e4e9fd5f1a4856edeb1f0fdcc5078ad4"
server_key_iv: "28f6cd46201048adf6a6ba3def2ff302"
server_key_version: "encryptionkey@0"
I20260928 03:09:27.091058 25552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:27.112421 25552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:27.117544 25552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:27.117879 25552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:27.118930 25552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:27.120994 25552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:27.121125 25552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:27.121275 25552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:27.121337 25552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:27.158207 25552 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:44549
I20260928 03:09:27.158275 25679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:44549 every 8 connection(s)
I20260928 03:09:27.159715 25552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:09:27.167078 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 25552
I20260928 03:09:27.167302 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:09:27.167932 6912 external_mini_cluster.cc:1428] Setting key cec3d77530627cc7c13525f6ef2da0fe
Sep 28 03:09:27 dist-test-slave-fjww krb5kdc[25178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564967, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:09:27.184048 25682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.162660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:37239 (local address 127.6.192.62:41207)
0928 03:09:27.163161 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:27.163165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:27.164288 (+ 1123us) server_negotiation.cc:407] Connection header received
0928 03:09:27.166201 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:27.166207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:27.166316 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:27.166440 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:27.168781 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.169844 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.172147 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.172636 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.172784 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.177021 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:27.177028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:27.177032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:27.177073 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:27.180304 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:27.181324 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:27.181341 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:27.181344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:27.181487 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:27.182003 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:27.182008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:27.182010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:27.182328 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:27.182526 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:27.183331 (+ 805us) server_negotiation.cc:299] Negotiation successful
0928 03:09:27.183632 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":260,"threads_started":1}
I20260928 03:09:27.185132 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.163213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:41207 (local address 127.6.192.3:37239)
0928 03:09:27.164097 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:27.164144 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:27.165895 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:27.166817 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:27.166833 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:27.167687 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:27.168572 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:27.168595 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.170017 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.170021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:27.171962 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:27.171982 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.173017 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.173025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:27.173248 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.174028 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:09:27.174059 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:27.176595 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:27.180509 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:27.180527 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:27.180543 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:27.181156 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:27.181615 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:27.181621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:27.181624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:27.181857 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:27.182501 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:27.182510 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:27.182936 (+ 426us) client_negotiation.cc:769] Sending connection context
0928 03:09:27.183230 (+ 294us) client_negotiation.cc:240] Negotiation successful
0928 03:09:27.183897 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":338,"threads_started":1}
I20260928 03:09:27.187110 25680 heartbeater.cc:347] Connected to a master server at 127.6.192.62:41207
I20260928 03:09:27.188047 25680 heartbeater.cc:464] Registering TS with master...
I20260928 03:09:27.189188 25680 heartbeater.cc:511] Master 127.6.192.62:41207 requested a full tablet report, sending...
I20260928 03:09:27.191489 25223 ts_manager.cc:194] Registered new tserver with Master: dec87d0d43624986a1a79ac24f62220c (127.6.192.3:44549)
I20260928 03:09:27.193013 25223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:37239
I20260928 03:09:27.198297 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:09:27.221354 25682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.203035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46998 (local address 127.6.192.62:41207)
0928 03:09:27.203280 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:27.203289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:27.203793 (+ 504us) server_negotiation.cc:407] Connection header received
0928 03:09:27.203965 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:27.203974 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:27.204078 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:27.204211 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:27.205820 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.207219 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.208703 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.209547 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.209702 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.212620 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:27.212629 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:27.212637 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:27.212697 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:27.217112 (+ 4415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:27.218313 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:27.218322 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:27.218327 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:27.218529 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:27.219236 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:27.219244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:27.219250 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:27.219822 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:27.220158 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:27.220636 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 03:09:27.220894 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 03:09:27.225867 25223 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:09:27.233220 25223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:09:27.285373 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.246934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:35399 (local address 127.0.0.1:55934)
0928 03:09:27.273495 (+ 26561us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:27.273552 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:27.273779 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:27.276531 (+ 2752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:27.276539 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:27.276579 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:27.277116 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:27.277134 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.279438 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.279483 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:27.282092 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:27.282105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.284312 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.284320 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:27.284510 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.284580 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 03:09:27.284710 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 03:09:27.284954 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26190,"thread_start_us":28598,"threads_started":1}
I20260928 03:09:27.287240 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.247661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55934 (local address 127.6.192.1:35399)
0928 03:09:27.274550 (+ 26889us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:27.274563 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:27.274603 (+ 40us) server_negotiation.cc:407] Connection header received
0928 03:09:27.274734 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:27.274746 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:27.275109 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:27.275342 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:27.277855 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.279278 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.282289 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.284061 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.286175 (+ 2114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.286534 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:27.286640 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 03:09:27.286872 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26685,"thread_start_us":26521,"threads_started":1}
I20260928 03:09:27.287842 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.246946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:44549 (local address 127.0.0.1:42144)
0928 03:09:27.276317 (+ 29371us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:27.276338 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:27.276464 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:27.279530 (+ 3066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:27.279539 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:27.279568 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:27.280015 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:27.280027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.282511 (+ 2484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.282516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:27.284697 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:27.284712 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.286910 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.286917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:27.287105 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.287139 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 03:09:27.287218 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 03:09:27.287400 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29230,"mutex_wait_us":1,"thread_start_us":29775,"threads_started":1}
I20260928 03:09:27.289242 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.247544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42144 (local address 127.6.192.3:44549)
0928 03:09:27.276388 (+ 28844us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:27.276396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:27.276429 (+ 33us) server_negotiation.cc:407] Connection header received
0928 03:09:27.277370 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:27.277379 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:27.277633 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:27.280221 (+ 2588us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:27.280519 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.282089 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.284922 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.286677 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.288216 (+ 1539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.288486 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:27.288571 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 03:09:27.288765 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28665,"thread_start_us":28160,"threads_started":1}
I20260928 03:09:27.289719 25693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.247419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:42333 (local address 127.0.0.1:54644)
0928 03:09:27.277054 (+ 29635us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:27.277076 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:27.277200 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:27.278173 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:27.278177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:27.278202 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:27.278550 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:27.278561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.281078 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.281086 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:27.283954 (+ 2868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:27.283975 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.288870 (+ 4895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.288877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:27.289090 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.289123 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 03:09:27.289235 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 03:09:27.289372 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29483,"thread_start_us":34745,"threads_started":1}
I20260928 03:09:27.290473 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.248044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.6.192.2:42333)
0928 03:09:27.272570 (+ 24526us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:27.272584 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:27.277329 (+ 4745us) server_negotiation.cc:407] Connection header received
0928 03:09:27.277429 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:27.277435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:27.277632 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:27.277849 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:27.278769 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.280881 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.284205 (+ 3324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.288218 (+ 4013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.288385 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.288691 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:27.289619 (+ 928us) server_negotiation.cc:299] Negotiation successful
0928 03:09:27.290010 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24269,"thread_start_us":24550,"threads_started":1}
I20260928 03:09:27.292150 25347 tablet_service.cc:1467] Processing CreateTablet for tablet 1764cd34d8cf47f58d680d2cd46ec817 (DEFAULT_TABLE table=test-table [id=872c1f3775f34a6a8a6205f910fd7c89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:09:27.294025 25347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1764cd34d8cf47f58d680d2cd46ec817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:27.294746 25481 tablet_service.cc:1467] Processing CreateTablet for tablet 1764cd34d8cf47f58d680d2cd46ec817 (DEFAULT_TABLE table=test-table [id=872c1f3775f34a6a8a6205f910fd7c89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:09:27.293550 25615 tablet_service.cc:1467] Processing CreateTablet for tablet 1764cd34d8cf47f58d680d2cd46ec817 (DEFAULT_TABLE table=test-table [id=872c1f3775f34a6a8a6205f910fd7c89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:09:27.296697 25481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1764cd34d8cf47f58d680d2cd46ec817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:27.296883 25615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1764cd34d8cf47f58d680d2cd46ec817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:27.317361 25699 tablet_bootstrap.cc:492] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976: Bootstrap starting.
I20260928 03:09:27.322073 25699 tablet_bootstrap.cc:654] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:27.325032 25700 tablet_bootstrap.cc:492] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992: Bootstrap starting.
I20260928 03:09:27.325109 25699 log.cc:824] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:27.328377 25701 tablet_bootstrap.cc:492] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c: Bootstrap starting.
I20260928 03:09:27.333853 25700 tablet_bootstrap.cc:654] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:27.336160 25700 log.cc:824] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:27.336951 25701 tablet_bootstrap.cc:654] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:27.338330 25701 log.cc:824] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:27.340155 25699 tablet_bootstrap.cc:492] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976: No bootstrap required, opened a new log
I20260928 03:09:27.340595 25699 ts_tablet_manager.cc:1397] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.006s
I20260928 03:09:27.355288 25701 tablet_bootstrap.cc:492] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c: No bootstrap required, opened a new log
I20260928 03:09:27.355701 25701 ts_tablet_manager.cc:1397] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.013s
I20260928 03:09:27.359536 25699 raft_consensus.cc:348] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec87d0d43624986a1a79ac24f62220c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44549 } } peers { permanent_uuid: "8d47b979fa98482db49690c94c1db976" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 35399 } } peers { permanent_uuid: "736d050cbd114d138b002eddcbaf0992" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42333 } }
I20260928 03:09:27.360077 25699 raft_consensus.cc:374] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:27.360183 25699 raft_consensus.cc:729] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d47b979fa98482db49690c94c1db976, State: Initialized, Role: FOLLOWER
I20260928 03:09:27.361011 25699 consensus_queue.cc:260] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec87d0d43624986a1a79ac24f62220c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44549 } } peers { permanent_uuid: "8d47b979fa98482db49690c94c1db976" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 35399 } } peers { permanent_uuid: "736d050cbd114d138b002eddcbaf0992" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42333 } }
I20260928 03:09:27.363174 25700 tablet_bootstrap.cc:492] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992: No bootstrap required, opened a new log
I20260928 03:09:27.363663 25700 ts_tablet_manager.cc:1397] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992: Time spent bootstrapping tablet: real 0.039s user 0.011s sys 0.007s
I20260928 03:09:27.369334 25701 raft_consensus.cc:348] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec87d0d43624986a1a79ac24f62220c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44549 } } peers { permanent_uuid: "8d47b979fa98482db49690c94c1db976" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 35399 } } peers { permanent_uuid: "736d050cbd114d138b002eddcbaf0992" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42333 } }
I20260928 03:09:27.370039 25701 raft_consensus.cc:374] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:27.370187 25701 raft_consensus.cc:729] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec87d0d43624986a1a79ac24f62220c, State: Initialized, Role: FOLLOWER
I20260928 03:09:27.371786 25701 consensus_queue.cc:260] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec87d0d43624986a1a79ac24f62220c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44549 } } peers { permanent_uuid: "8d47b979fa98482db49690c94c1db976" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 35399 } } peers { permanent_uuid: "736d050cbd114d138b002eddcbaf0992" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42333 } }
I20260928 03:09:27.372350 25699 ts_tablet_manager.cc:1428] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976: Time spent starting tablet: real 0.032s user 0.021s sys 0.006s
I20260928 03:09:27.378556 25701 ts_tablet_manager.cc:1428] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c: Time spent starting tablet: real 0.023s user 0.009s sys 0.011s
I20260928 03:09:27.378984 25680 heartbeater.cc:503] Master 127.6.192.62:41207 was elected leader, sending a full tablet report...
I20260928 03:09:27.389181 25700 raft_consensus.cc:348] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec87d0d43624986a1a79ac24f62220c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44549 } } peers { permanent_uuid: "8d47b979fa98482db49690c94c1db976" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 35399 } } peers { permanent_uuid: "736d050cbd114d138b002eddcbaf0992" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42333 } }
I20260928 03:09:27.389906 25700 raft_consensus.cc:374] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:27.390074 25700 raft_consensus.cc:729] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 736d050cbd114d138b002eddcbaf0992, State: Initialized, Role: FOLLOWER
I20260928 03:09:27.390946 25700 consensus_queue.cc:260] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec87d0d43624986a1a79ac24f62220c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44549 } } peers { permanent_uuid: "8d47b979fa98482db49690c94c1db976" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 35399 } } peers { permanent_uuid: "736d050cbd114d138b002eddcbaf0992" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42333 } }
I20260928 03:09:27.392453 25546 heartbeater.cc:503] Master 127.6.192.62:41207 was elected leader, sending a full tablet report...
I20260928 03:09:27.393527 25700 ts_tablet_manager.cc:1428] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992: Time spent starting tablet: real 0.030s user 0.015s sys 0.004s
W20260928 03:09:27.411634 25681 tablet.cc:2369] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:09:27.473520 25706 raft_consensus.cc:482] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:09:27.473843 25706 raft_consensus.cc:504] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec87d0d43624986a1a79ac24f62220c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44549 } } peers { permanent_uuid: "8d47b979fa98482db49690c94c1db976" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 35399 } } peers { permanent_uuid: "736d050cbd114d138b002eddcbaf0992" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42333 } }
I20260928 03:09:27.475769 25706 leader_election.cc:288] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d47b979fa98482db49690c94c1db976 (127.6.192.1:35399), 736d050cbd114d138b002eddcbaf0992 (127.6.192.2:42333)
I20260928 03:09:27.481496 25707 raft_consensus.cc:482] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:09:27.482057 25707 raft_consensus.cc:504] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec87d0d43624986a1a79ac24f62220c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44549 } } peers { permanent_uuid: "8d47b979fa98482db49690c94c1db976" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 35399 } } peers { permanent_uuid: "736d050cbd114d138b002eddcbaf0992" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42333 } }
I20260928 03:09:27.484891 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.476149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:42333 (local address 127.6.192.3:44793)
0928 03:09:27.476341 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:27.476370 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:27.476475 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:27.477438 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:27.477443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:27.477478 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:27.477684 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:27.477697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.479025 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.479033 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:27.480986 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:27.481001 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.482900 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.482906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:27.484197 (+ 1291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.484240 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 03:09:27.484345 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 03:09:27.484556 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":52}
I20260928 03:09:27.485342 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.476255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:51541 (local address 127.6.192.1:35399)
0928 03:09:27.476718 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:27.476725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:27.477534 (+ 809us) server_negotiation.cc:407] Connection header received
0928 03:09:27.477810 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:27.477820 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:27.477924 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:27.478033 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:27.478642 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.480030 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.482588 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.484143 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.484253 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.484372 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:27.484968 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 03:09:27.485110 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277}
I20260928 03:09:27.485615 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.476800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:44793 (local address 127.6.192.2:42333)
0928 03:09:27.477028 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:27.477038 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:27.477062 (+ 24us) server_negotiation.cc:407] Connection header received
0928 03:09:27.477158 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:27.477167 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:27.477276 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:27.477400 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:27.477855 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.478900 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.481200 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.482565 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.482701 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.482812 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:27.485119 (+ 2307us) server_negotiation.cc:299] Negotiation successful
0928 03:09:27.485263 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 03:09:27.485988 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.476179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:35399 (local address 127.6.192.3:51541)
0928 03:09:27.477441 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:27.477457 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:27.477630 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:27.478061 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:27.478066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:27.478093 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:27.478463 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:27.478474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.480195 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.480201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:27.482390 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:27.482406 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.484567 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.484572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:27.484758 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.484788 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 03:09:27.485523 (+ 735us) client_negotiation.cc:240] Negotiation successful
0928 03:09:27.485684 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1123,"mutex_wait_us":435,"thread_start_us":501,"threads_started":1}
I20260928 03:09:27.487041 25707 leader_election.cc:288] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec87d0d43624986a1a79ac24f62220c (127.6.192.3:44549), 8d47b979fa98482db49690c94c1db976 (127.6.192.1:35399)
I20260928 03:09:27.486881 25501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1764cd34d8cf47f58d680d2cd46ec817" candidate_uuid: "dec87d0d43624986a1a79ac24f62220c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "736d050cbd114d138b002eddcbaf0992" is_pre_election: true
I20260928 03:09:27.487628 25501 raft_consensus.cc:2454] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec87d0d43624986a1a79ac24f62220c in term 0.
I20260928 03:09:27.488972 25367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1764cd34d8cf47f58d680d2cd46ec817" candidate_uuid: "dec87d0d43624986a1a79ac24f62220c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d47b979fa98482db49690c94c1db976" is_pre_election: true
I20260928 03:09:27.489723 25367 raft_consensus.cc:2454] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec87d0d43624986a1a79ac24f62220c in term 0.
I20260928 03:09:27.490728 25571 leader_election.cc:302] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d47b979fa98482db49690c94c1db976, dec87d0d43624986a1a79ac24f62220c; no voters:
I20260928 03:09:27.491318 25706 raft_consensus.cc:2781] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:09:27.491583 25706 raft_consensus.cc:482] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:09:27.491726 25706 raft_consensus.cc:3037] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:27.498631 25706 raft_consensus.cc:504] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec87d0d43624986a1a79ac24f62220c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44549 } } peers { permanent_uuid: "8d47b979fa98482db49690c94c1db976" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 35399 } } peers { permanent_uuid: "736d050cbd114d138b002eddcbaf0992" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42333 } }
I20260928 03:09:27.499729 25706 leader_election.cc:288] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [CANDIDATE]: Term 1 election: Requested vote from peers 8d47b979fa98482db49690c94c1db976 (127.6.192.1:35399), 736d050cbd114d138b002eddcbaf0992 (127.6.192.2:42333)
I20260928 03:09:27.501070 25501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1764cd34d8cf47f58d680d2cd46ec817" candidate_uuid: "dec87d0d43624986a1a79ac24f62220c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "736d050cbd114d138b002eddcbaf0992"
I20260928 03:09:27.501329 25501 raft_consensus.cc:3037] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:27.501305 25367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1764cd34d8cf47f58d680d2cd46ec817" candidate_uuid: "dec87d0d43624986a1a79ac24f62220c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d47b979fa98482db49690c94c1db976"
I20260928 03:09:27.501591 25367 raft_consensus.cc:3037] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:27.504642 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.487561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:33061 (local address 127.6.192.1:35399)
0928 03:09:27.487772 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:27.487780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:27.491190 (+ 3410us) server_negotiation.cc:407] Connection header received
0928 03:09:27.491841 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:27.491851 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:27.491957 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:27.492042 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:27.493246 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.494988 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.498510 (+ 3522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.500023 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.500135 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.500321 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:27.504158 (+ 3837us) server_negotiation.cc:299] Negotiation successful
0928 03:09:27.504329 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 03:09:27.505162 25501 raft_consensus.cc:2454] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec87d0d43624986a1a79ac24f62220c in term 1.
I20260928 03:09:27.505148 25367 raft_consensus.cc:2454] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec87d0d43624986a1a79ac24f62220c in term 1.
I20260928 03:09:27.505486 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.487371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:35399 (local address 127.6.192.2:33061)
0928 03:09:27.491059 (+ 3688us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:27.491091 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:27.491379 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:27.492336 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:27.492342 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:27.492373 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:27.492981 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:27.493001 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.495271 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.495280 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:27.498264 (+ 2984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:27.498292 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.503720 (+ 5428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.503727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:27.503946 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.503985 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 03:09:27.504842 (+ 857us) client_negotiation.cc:240] Negotiation successful
0928 03:09:27.505033 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3431,"mutex_wait_us":1,"thread_start_us":12558,"threads_started":1}
I20260928 03:09:27.505679 25571 leader_election.cc:302] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d47b979fa98482db49690c94c1db976, dec87d0d43624986a1a79ac24f62220c; no voters:
I20260928 03:09:27.506057 25706 raft_consensus.cc:2781] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:27.506227 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1764cd34d8cf47f58d680d2cd46ec817" candidate_uuid: "736d050cbd114d138b002eddcbaf0992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d47b979fa98482db49690c94c1db976" is_pre_election: true
I20260928 03:09:27.506676 25366 raft_consensus.cc:2379] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 736d050cbd114d138b002eddcbaf0992 in current term 1: Already voted for candidate dec87d0d43624986a1a79ac24f62220c in this term.
I20260928 03:09:27.506747 25706 raft_consensus.cc:686] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [term 1 LEADER]: Becoming Leader. State: Replica: dec87d0d43624986a1a79ac24f62220c, State: Running, Role: LEADER
I20260928 03:09:27.507418 25706 consensus_queue.cc:237] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec87d0d43624986a1a79ac24f62220c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44549 } } peers { permanent_uuid: "8d47b979fa98482db49690c94c1db976" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 35399 } } peers { permanent_uuid: "736d050cbd114d138b002eddcbaf0992" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42333 } }
I20260928 03:09:27.509402 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.488593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:38421 (local address 127.6.192.3:44549)
0928 03:09:27.488827 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:27.488834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:27.501001 (+ 12167us) server_negotiation.cc:407] Connection header received
0928 03:09:27.501260 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:27.501267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:27.501358 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:27.501467 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:27.502448 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.503590 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.506831 (+ 3241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.508267 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.508379 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.508502 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:27.509119 (+ 617us) server_negotiation.cc:299] Negotiation successful
0928 03:09:27.509222 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 03:09:27.509617 25710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.488248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:44549 (local address 127.6.192.2:38421)
0928 03:09:27.500880 (+ 12632us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:27.500900 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:27.501098 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:27.501573 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:27.501579 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:27.501607 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:27.502197 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:27.502212 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.503790 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.503797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:27.506576 (+ 2779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:27.506595 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.508741 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.508752 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:27.508940 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.508979 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 03:09:27.509096 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 03:09:27.509243 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12447,"thread_start_us":12216,"threads_started":1}
I20260928 03:09:27.510747 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1764cd34d8cf47f58d680d2cd46ec817" candidate_uuid: "736d050cbd114d138b002eddcbaf0992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec87d0d43624986a1a79ac24f62220c" is_pre_election: true
I20260928 03:09:27.512019 25435 leader_election.cc:302] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 736d050cbd114d138b002eddcbaf0992; no voters: 8d47b979fa98482db49690c94c1db976, dec87d0d43624986a1a79ac24f62220c
I20260928 03:09:27.512506 25707 raft_consensus.cc:2726] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 03:09:27.513226 25223 catalog_manager.cc:5495] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c reported cstate change: term changed from 0 to 1, leader changed from <none> to dec87d0d43624986a1a79ac24f62220c (127.6.192.3). New cstate: current_term: 1 leader_uuid: "dec87d0d43624986a1a79ac24f62220c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec87d0d43624986a1a79ac24f62220c" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 44549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d47b979fa98482db49690c94c1db976" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 35399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "736d050cbd114d138b002eddcbaf0992" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42333 } health_report { overall_health: UNKNOWN } } }
W20260928 03:09:27.550757 25413 tablet.cc:2369] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:09:27.568392 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.562224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.6.192.3:44549)
0928 03:09:27.562464 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:27.562470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:27.562589 (+ 119us) server_negotiation.cc:407] Connection header received
0928 03:09:27.562928 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:27.562932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:27.563033 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:27.563198 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:09:27.563732 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.564560 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.566018 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.566428 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.566565 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.567131 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:09:27.567722 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:09:27.567928 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:27.568057 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 03:09:27.568118 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 03:09:27.577131 25367 raft_consensus.cc:1261] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976 [term 1 FOLLOWER]: Refusing update from remote peer dec87d0d43624986a1a79ac24f62220c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:09:27.577193 25501 raft_consensus.cc:1261] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992 [term 1 FOLLOWER]: Refusing update from remote peer dec87d0d43624986a1a79ac24f62220c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:09:27.578166 25706 consensus_queue.cc:1035] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [LEADER]: Connected to new peer: Peer: permanent_uuid: "736d050cbd114d138b002eddcbaf0992" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 42333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:09:27.578504 25711 consensus_queue.cc:1035] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d47b979fa98482db49690c94c1db976" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 35399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:09:27.591652 25714 mvcc.cc:205] Tried to move back new op lower bound from 7334154107184750592 to 7334154106915987456. Current Snapshot: MvccSnapshot[applied={T|T < 7334154107184750592}]
I20260928 03:09:27.594919 25715 mvcc.cc:205] Tried to move back new op lower bound from 7334154107184750592 to 7334154106915987456. Current Snapshot: MvccSnapshot[applied={T|T < 7334154107184750592}]
I20260928 03:09:27.610054 25222 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:09:27.610442 25222 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:09:27.614845 25222 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 856785a6970c4768879d349de9fa559b: Sending DeleteTablet for 3 replicas of tablet 1764cd34d8cf47f58d680d2cd46ec817
I20260928 03:09:27.616134 25615 tablet_service.cc:1514] Processing DeleteTablet for tablet 1764cd34d8cf47f58d680d2cd46ec817 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:09:27 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:42144
I20260928 03:09:27.616734 25347 tablet_service.cc:1514] Processing DeleteTablet for tablet 1764cd34d8cf47f58d680d2cd46ec817 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:09:27 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:55934
I20260928 03:09:27.619091 25481 tablet_service.cc:1514] Processing DeleteTablet for tablet 1764cd34d8cf47f58d680d2cd46ec817 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:09:27 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:54644
I20260928 03:09:27.628041 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 25284
I20260928 03:09:27.641939 25725 tablet_replica.cc:330] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992: stopping tablet replica
I20260928 03:09:27.642683 25725 raft_consensus.cc:2229] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:09:27.643392 25725 raft_consensus.cc:2258] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:09:27.644344 25724 tablet_replica.cc:330] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976: stopping tablet replica
I20260928 03:09:27.646123 25724 raft_consensus.cc:2229] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:09:27.646675 25724 raft_consensus.cc:2258] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:09:27.647058 25723 tablet_replica.cc:330] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c: stopping tablet replica
I20260928 03:09:27.646955 25725 ts_tablet_manager.cc:1905] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:09:27.647503 25723 raft_consensus.cc:2229] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:09:27.648371 25723 raft_consensus.cc:2258] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:09:27.650508 25723 ts_tablet_manager.cc:1905] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:09:27.651113 25724 ts_tablet_manager.cc:1905] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:09:27.657140 25725 ts_tablet_manager.cc:1918] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:09:27.657330 25725 log.cc:1196] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/1764cd34d8cf47f58d680d2cd46ec817
I20260928 03:09:27.657855 25725 ts_tablet_manager.cc:1939] T 1764cd34d8cf47f58d680d2cd46ec817 P 736d050cbd114d138b002eddcbaf0992: Deleting consensus metadata
I20260928 03:09:27.658607 25723 ts_tablet_manager.cc:1918] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:09:27.658854 25723 log.cc:1196] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/1764cd34d8cf47f58d680d2cd46ec817
I20260928 03:09:27.659256 25724 ts_tablet_manager.cc:1918] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:09:27.659459 25724 log.cc:1196] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/1764cd34d8cf47f58d680d2cd46ec817
I20260928 03:09:27.659682 25723 ts_tablet_manager.cc:1939] T 1764cd34d8cf47f58d680d2cd46ec817 P dec87d0d43624986a1a79ac24f62220c: Deleting consensus metadata
I20260928 03:09:27.660012 25724 ts_tablet_manager.cc:1939] T 1764cd34d8cf47f58d680d2cd46ec817 P 8d47b979fa98482db49690c94c1db976: Deleting consensus metadata
I20260928 03:09:27.660426 25207 catalog_manager.cc:4841] TS 736d050cbd114d138b002eddcbaf0992 (127.6.192.2:42333): tablet 1764cd34d8cf47f58d680d2cd46ec817 (table test-table [id=872c1f3775f34a6a8a6205f910fd7c89]) successfully deleted
I20260928 03:09:27.661793 25208 catalog_manager.cc:4841] TS dec87d0d43624986a1a79ac24f62220c (127.6.192.3:44549): tablet 1764cd34d8cf47f58d680d2cd46ec817 (table test-table [id=872c1f3775f34a6a8a6205f910fd7c89]) successfully deleted
I20260928 03:09:27.661821 25210 catalog_manager.cc:4841] TS 8d47b979fa98482db49690c94c1db976 (127.6.192.1:35399): tablet 1764cd34d8cf47f58d680d2cd46ec817 (table test-table [id=872c1f3775f34a6a8a6205f910fd7c89]) successfully deleted
Sep 28 03:09:27 dist-test-slave-fjww krb5kdc[25178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:09:27.672556 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.630535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.6.192.1:35399)
0928 03:09:27.630724 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:27.630730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:27.645377 (+ 14647us) server_negotiation.cc:407] Connection header received
0928 03:09:27.645670 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:27.645679 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:27.645810 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:27.645913 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:27.647923 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.648978 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.650963 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.651682 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.651784 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.665390 (+ 13606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:27.665405 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:27.665420 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:27.665461 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:27.668734 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:27.669634 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:27.669642 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:27.669644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:27.669893 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:27.670382 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:27.670387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:27.670390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:27.670750 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:27.671057 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:27.671821 (+ 764us) server_negotiation.cc:299] Negotiation successful
0928 03:09:27.672096 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 03:09:27.673323 25407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:27.781160 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 25284
I20260928 03:09:27.806190 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 25418
Sep 28 03:09:27 dist-test-slave-fjww krb5kdc[25178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.2@KRBTEST.COM
I20260928 03:09:27.825978 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.807007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54652 (local address 127.6.192.2:42333)
0928 03:09:27.807188 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:27.807193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:27.807285 (+ 92us) server_negotiation.cc:407] Connection header received
0928 03:09:27.807535 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:27.807539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:27.807606 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:27.807825 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:27.809644 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.810482 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.812568 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.813080 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.813195 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.818971 (+ 5776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:27.818990 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:27.819007 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:27.819052 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:27.822552 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:27.823492 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:27.823505 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:27.823512 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:27.823674 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:27.824162 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:27.824168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:27.824173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:27.824571 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:27.824933 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:27.825263 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 03:09:27.825536 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 03:09:27.826814 25541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:27.922904 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 25418
I20260928 03:09:27.947144 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 25552
Sep 28 03:09:27 dist-test-slave-fjww krb5kdc[25178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.3@KRBTEST.COM
I20260928 03:09:27.966960 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:27.948214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42158 (local address 127.6.192.3:44549)
0928 03:09:27.948434 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:27.948441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:27.948466 (+ 25us) server_negotiation.cc:407] Connection header received
0928 03:09:27.948671 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:27.948677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:27.948824 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:27.948949 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:27.950380 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.951718 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.953107 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:27.953809 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:27.953978 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:27.958850 (+ 4872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:27.958880 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:27.958905 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:27.958976 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:27.963524 (+ 4548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:27.964487 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:27.964498 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:27.964502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:27.964684 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:27.965190 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:27.965195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:27.965200 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:27.965682 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:27.966017 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:27.966319 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 03:09:27.966582 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 03:09:27.967922 25675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:28.092325 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 25552
I20260928 03:09:28.112458 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 25191
I20260928 03:09:28.113735 25253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:28.239952 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 25191
2026-09-28T03:09:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4808 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T03:09:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:09:28Z Disabled control of system clock
I20260928 03:09:28.326007 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:35561
--webserver_interface=127.6.192.62
--webserver_port=0
--builtin_ntp_servers=127.6.192.20:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:35561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 03:09:28.686054 25734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:28.686376 25734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:28.686462 25734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:28.694352 25734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:09:28.694489 25734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:28.694527 25734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:28.694581 25734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:09:28.694626 25734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:09:28.704324 25734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:35561
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:35561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:28.705663 25734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:28.707634 25734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:28.720530 25742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:28.720664 25740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:28.720705 25739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:28.722295 25734 server_base.cc:1034] running on GCE node
I20260928 03:09:28.723263 25734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:28.724926 25734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:28.726186 25734 hybrid_clock.cc:648] HybridClock initialized: now 1790564968726191 us; error 119 us; skew 500 ppm
I20260928 03:09:28.726580 25734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:09:28.729332 25734 webserver.cc:460] Webserver started at http://127.6.192.62:33563/ using document root <none> and password file <none>
I20260928 03:09:28.730098 25734 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:28.730242 25734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:28.730572 25734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:28.733819 25734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "9d8834e0bc34441385bb0622d868a487"
format_stamp: "Formatted at 2026-09-28 03:09:28 on dist-test-slave-fjww"
server_key: "a3761bdf3b43c522195e0a9c8493c146"
server_key_iv: "2814428592ef933b516810e82d31aab4"
server_key_version: "encryptionkey@0"
I20260928 03:09:28.734726 25734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "9d8834e0bc34441385bb0622d868a487"
format_stamp: "Formatted at 2026-09-28 03:09:28 on dist-test-slave-fjww"
server_key: "a3761bdf3b43c522195e0a9c8493c146"
server_key_iv: "2814428592ef933b516810e82d31aab4"
server_key_version: "encryptionkey@0"
I20260928 03:09:28.740523 25734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 03:09:28.744699 25748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:28.746405 25734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 03:09:28.746654 25734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "9d8834e0bc34441385bb0622d868a487"
format_stamp: "Formatted at 2026-09-28 03:09:28 on dist-test-slave-fjww"
server_key: "a3761bdf3b43c522195e0a9c8493c146"
server_key_iv: "2814428592ef933b516810e82d31aab4"
server_key_version: "encryptionkey@0"
I20260928 03:09:28.746942 25734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:28.786216 25734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:28.787269 25734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:28.787652 25734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:28.814384 25734 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:35561
I20260928 03:09:28.814394 25799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:35561 every 8 connection(s)
I20260928 03:09:28.816059 25734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:09:28.820592 25800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:28.825462 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 25734
I20260928 03:09:28.825654 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:09:28.826176 6912 external_mini_cluster.cc:1428] Setting key 895c31f51169ef08337420b6aeb9eb6c
I20260928 03:09:28.834630 25800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487: Bootstrap starting.
I20260928 03:09:28.838477 25800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:28.839911 25800 log.cc:824] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:28.843201 25800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487: No bootstrap required, opened a new log
I20260928 03:09:28.843365 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:28.828409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50416 (local address 127.6.192.62:35561)
0928 03:09:28.829580 (+ 1171us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:28.829609 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:28.829665 (+ 56us) server_negotiation.cc:407] Connection header received
0928 03:09:28.832635 (+ 2970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:28.832672 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:28.833375 (+ 703us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:28.833863 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:28.834465 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:28.836519 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:28.838231 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:28.839335 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:28.839564 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:28.840951 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:28.840962 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:28.840977 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:28.841550 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:28.841629 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:28.841650 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:28.841782 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:28.842042 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 03:09:28.842368 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":376,"threads_started":1}
I20260928 03:09:28.852852 25800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8834e0bc34441385bb0622d868a487" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 35561 } }
I20260928 03:09:28.853278 25800 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:28.853422 25800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d8834e0bc34441385bb0622d868a487, State: Initialized, Role: FOLLOWER
I20260928 03:09:28.854054 25800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8834e0bc34441385bb0622d868a487" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 35561 } }
I20260928 03:09:28.854290 25800 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:09:28.854460 25800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:09:28.854664 25800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:28.857285 25800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8834e0bc34441385bb0622d868a487" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 35561 } }
I20260928 03:09:28.857940 25800 leader_election.cc:302] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d8834e0bc34441385bb0622d868a487; no voters:
I20260928 03:09:28.858712 25800 leader_election.cc:288] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:09:28.858909 25805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:28.859862 25805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487 [term 1 LEADER]: Becoming Leader. State: Replica: 9d8834e0bc34441385bb0622d868a487, State: Running, Role: LEADER
I20260928 03:09:28.860378 25805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8834e0bc34441385bb0622d868a487" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 35561 } }
I20260928 03:09:28.861243 25800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:09:28.863366 25807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d8834e0bc34441385bb0622d868a487. Latest consensus state: current_term: 1 leader_uuid: "9d8834e0bc34441385bb0622d868a487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8834e0bc34441385bb0622d868a487" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 35561 } } }
I20260928 03:09:28.863804 25807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487 [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:28.866307 25806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d8834e0bc34441385bb0622d868a487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8834e0bc34441385bb0622d868a487" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 35561 } } }
I20260928 03:09:28.866703 25806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487 [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:28.867930 25811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:09:28.874135 25811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:09:28.885321 25811 catalog_manager.cc:1347] Generated new cluster ID: 766b67cb08954d3ba24cdeeb4ec439f7
I20260928 03:09:28.885450 25811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:09:28.912942 25811 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:09:28.914856 25811 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:09:28.928408 25811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487: Generated new TSK 0
I20260928 03:09:28.929400 25811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:09:28.935388 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:35561
--builtin_ntp_servers=127.6.192.20:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 03:09:29.293129 25824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:29.293488 25824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:29.293644 25824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:29.301316 25824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:29.301474 25824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:29.301652 25824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:09:29.311374 25824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:35561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:29.312906 25824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:29.314757 25824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:29.329371 25830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:29.329312 25832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:29.330058 25829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:29.330551 25824 server_base.cc:1034] running on GCE node
I20260928 03:09:29.331284 25824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:29.332476 25824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:29.333730 25824 hybrid_clock.cc:648] HybridClock initialized: now 1790564969333686 us; error 31 us; skew 500 ppm
I20260928 03:09:29.334162 25824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:09:29.337432 25824 webserver.cc:460] Webserver started at http://127.6.192.1:37093/ using document root <none> and password file <none>
I20260928 03:09:29.338341 25824 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:29.338438 25824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:29.338766 25824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:29.341775 25824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "5e758be175c046a8a48953fd2f59069d"
format_stamp: "Formatted at 2026-09-28 03:09:29 on dist-test-slave-fjww"
server_key: "215a54fbd82aa299b4adb00db8cb4ef9"
server_key_iv: "bf9361ca86a9781dfde74f316b4661d7"
server_key_version: "encryptionkey@0"
I20260928 03:09:29.342598 25824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "5e758be175c046a8a48953fd2f59069d"
format_stamp: "Formatted at 2026-09-28 03:09:29 on dist-test-slave-fjww"
server_key: "215a54fbd82aa299b4adb00db8cb4ef9"
server_key_iv: "bf9361ca86a9781dfde74f316b4661d7"
server_key_version: "encryptionkey@0"
I20260928 03:09:29.348208 25824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 03:09:29.352303 25838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:29.354204 25824 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:09:29.354430 25824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "5e758be175c046a8a48953fd2f59069d"
format_stamp: "Formatted at 2026-09-28 03:09:29 on dist-test-slave-fjww"
server_key: "215a54fbd82aa299b4adb00db8cb4ef9"
server_key_iv: "bf9361ca86a9781dfde74f316b4661d7"
server_key_version: "encryptionkey@0"
I20260928 03:09:29.354679 25824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:29.378108 25824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:29.379177 25824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:29.379577 25824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:29.380694 25824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:29.382524 25824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:29.382614 25824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:29.382748 25824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:29.382833 25824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:29.418773 25824 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:41857
I20260928 03:09:29.418996 25950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:41857 every 8 connection(s)
I20260928 03:09:29.420306 25824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:09:29.423049 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 25824
I20260928 03:09:29.423240 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:09:29.423805 6912 external_mini_cluster.cc:1428] Setting key 0b707ed1f20088b39e879a2792e164d3
I20260928 03:09:29.427336 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:35561
--builtin_ntp_servers=127.6.192.20:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 03:09:29.436389 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:29.423119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:41411 (local address 127.6.192.62:35561)
0928 03:09:29.423720 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:29.423724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:29.425326 (+ 1602us) server_negotiation.cc:407] Connection header received
0928 03:09:29.427532 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:29.427540 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:29.427645 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:29.427772 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:29.429317 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:29.430601 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:29.432667 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:29.433512 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:29.433673 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:29.435244 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:29.435255 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:29.435263 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:29.435304 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:29.435366 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:29.435386 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:29.435516 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:29.435797 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 03:09:29.435928 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":338,"threads_started":1}
I20260928 03:09:29.437736 25954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:29.423813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:35561 (local address 127.6.192.1:41411)
0928 03:09:29.425136 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:29.425198 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:29.427195 (+ 1997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:29.428209 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:29.428219 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:29.428328 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:29.429110 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:29.429129 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:29.430817 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:29.430823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:29.432489 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:29.432504 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:29.433768 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:29.433774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:29.434020 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:29.434867 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:09:29.434870 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:29.434914 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:09:29.435009 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:29.435552 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:29.435562 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:29.435569 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 03:09:29.435733 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 03:09:29.436106 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":405,"threads_started":1}
I20260928 03:09:29.439666 25951 heartbeater.cc:347] Connected to a master server at 127.6.192.62:35561
I20260928 03:09:29.440037 25951 heartbeater.cc:464] Registering TS with master...
I20260928 03:09:29.440953 25951 heartbeater.cc:511] Master 127.6.192.62:35561 requested a full tablet report, sending...
I20260928 03:09:29.443526 25765 ts_manager.cc:194] Registered new tserver with Master: 5e758be175c046a8a48953fd2f59069d (127.6.192.1:41857)
I20260928 03:09:29.445794 25765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.192.1:41411
W20260928 03:09:29.771376 25955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:29.771724 25955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:29.771826 25955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:29.779336 25955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:29.779533 25955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:29.779733 25955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:09:29.789291 25955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:35561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:29.790637 25955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:29.792701 25955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:29.807518 25960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:29.807787 25963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:29.807971 25961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:29.809568 25955 server_base.cc:1034] running on GCE node
I20260928 03:09:29.810308 25955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:29.811580 25955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:29.812896 25955 hybrid_clock.cc:648] HybridClock initialized: now 1790564969812766 us; error 65 us; skew 500 ppm
I20260928 03:09:29.813297 25955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:09:29.816337 25955 webserver.cc:460] Webserver started at http://127.6.192.2:34957/ using document root <none> and password file <none>
I20260928 03:09:29.817230 25955 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:29.817361 25955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:29.817775 25955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:29.821916 25955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "7e6d242e1dc746a6ad352868dc412554"
format_stamp: "Formatted at 2026-09-28 03:09:29 on dist-test-slave-fjww"
server_key: "1a16c36d48855792e374ce7423fd0eb0"
server_key_iv: "26027003036f0f56e30c9e540fe1aa69"
server_key_version: "encryptionkey@0"
I20260928 03:09:29.822989 25955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "7e6d242e1dc746a6ad352868dc412554"
format_stamp: "Formatted at 2026-09-28 03:09:29 on dist-test-slave-fjww"
server_key: "1a16c36d48855792e374ce7423fd0eb0"
server_key_iv: "26027003036f0f56e30c9e540fe1aa69"
server_key_version: "encryptionkey@0"
I20260928 03:09:29.830005 25955 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260928 03:09:29.834753 25969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:29.836413 25955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:09:29.836565 25955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "7e6d242e1dc746a6ad352868dc412554"
format_stamp: "Formatted at 2026-09-28 03:09:29 on dist-test-slave-fjww"
server_key: "1a16c36d48855792e374ce7423fd0eb0"
server_key_iv: "26027003036f0f56e30c9e540fe1aa69"
server_key_version: "encryptionkey@0"
I20260928 03:09:29.836760 25955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:29.852182 25955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:29.853092 25955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:29.853389 25955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:29.854415 25955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:29.856033 25955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:29.856153 25955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:29.856282 25955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:29.856369 25955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:29.892925 25955 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:45367
I20260928 03:09:29.892926 26081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:45367 every 8 connection(s)
I20260928 03:09:29.894264 25955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:09:29.898432 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 25955
I20260928 03:09:29.898612 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:09:29.899215 6912 external_mini_cluster.cc:1428] Setting key 303ce94762af7db8c95ee45e09d7249a
I20260928 03:09:29.902308 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:35561
--builtin_ntp_servers=127.6.192.20:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 03:09:29.908110 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:29.897109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:49513 (local address 127.6.192.62:35561)
0928 03:09:29.897291 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:29.897296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:29.899001 (+ 1705us) server_negotiation.cc:407] Connection header received
0928 03:09:29.901302 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:29.901307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:29.901394 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:29.901512 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:29.902736 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:29.903531 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:29.905160 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:29.905563 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:29.905704 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:29.907247 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:29.907253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:29.907260 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:29.907283 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:29.907316 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:29.907328 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:29.907427 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:29.907688 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 03:09:29.907800 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 03:09:29.909204 26084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:29.897547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:35561 (local address 127.6.192.2:49513)
0928 03:09:29.898785 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:29.898845 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:29.900975 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:29.901764 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:29.901774 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:29.901871 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:29.902563 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:29.902591 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:29.903686 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:29.903691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:29.904984 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:29.904996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:29.905832 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:29.905842 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:29.906096 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:29.906889 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:09:29.906892 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:29.906935 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:09:29.907024 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:29.907428 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:29.907475 (+ 47us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:29.907483 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 03:09:29.907680 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 03:09:29.908113 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":378,"threads_started":1}
I20260928 03:09:29.910924 26082 heartbeater.cc:347] Connected to a master server at 127.6.192.62:35561
I20260928 03:09:29.911262 26082 heartbeater.cc:464] Registering TS with master...
I20260928 03:09:29.912257 26082 heartbeater.cc:511] Master 127.6.192.62:35561 requested a full tablet report, sending...
I20260928 03:09:29.914075 25765 ts_manager.cc:194] Registered new tserver with Master: 7e6d242e1dc746a6ad352868dc412554 (127.6.192.2:45367)
I20260928 03:09:29.915041 25765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.192.2:49513
W20260928 03:09:30.248409 26085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:30.248778 26085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:30.248934 26085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:30.256512 26085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:30.256672 26085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:30.256855 26085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:09:30.266628 26085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:35561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:30.268103 26085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:30.270133 26085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:30.284564 26090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:30.284475 26093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:30.284528 26091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:30.285960 26085 server_base.cc:1034] running on GCE node
I20260928 03:09:30.286736 26085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:30.287963 26085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:30.289170 26085 hybrid_clock.cc:648] HybridClock initialized: now 1790564970289104 us; error 66 us; skew 500 ppm
I20260928 03:09:30.289527 26085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:09:30.296831 26085 webserver.cc:460] Webserver started at http://127.6.192.3:38887/ using document root <none> and password file <none>
I20260928 03:09:30.297623 26085 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:30.297746 26085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:30.298112 26085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:30.301306 26085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "2d2e3d66d39d4414885ff03755e20f3a"
format_stamp: "Formatted at 2026-09-28 03:09:30 on dist-test-slave-fjww"
server_key: "c2ff0124297b4f2bc3f6b7362ae23c15"
server_key_iv: "f92f6177acbf9ee7cd472d06c4b02885"
server_key_version: "encryptionkey@0"
I20260928 03:09:30.302192 26085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "2d2e3d66d39d4414885ff03755e20f3a"
format_stamp: "Formatted at 2026-09-28 03:09:30 on dist-test-slave-fjww"
server_key: "c2ff0124297b4f2bc3f6b7362ae23c15"
server_key_iv: "f92f6177acbf9ee7cd472d06c4b02885"
server_key_version: "encryptionkey@0"
I20260928 03:09:30.307909 26085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 03:09:30.311920 26099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:30.313455 26085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:09:30.313630 26085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "2d2e3d66d39d4414885ff03755e20f3a"
format_stamp: "Formatted at 2026-09-28 03:09:30 on dist-test-slave-fjww"
server_key: "c2ff0124297b4f2bc3f6b7362ae23c15"
server_key_iv: "f92f6177acbf9ee7cd472d06c4b02885"
server_key_version: "encryptionkey@0"
I20260928 03:09:30.313854 26085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:30.332062 26085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:30.332944 26085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:30.333237 26085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:30.334372 26085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:30.336091 26085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:30.336237 26085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:30.336350 26085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:30.336472 26085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:30.384711 26085 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:46519
I20260928 03:09:30.384809 26211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:46519 every 8 connection(s)
I20260928 03:09:30.386117 26085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:09:30.391696 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 26085
I20260928 03:09:30.391878 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:09:30.392330 6912 external_mini_cluster.cc:1428] Setting key e8d52b0e03516501e9dc9d1c00c8163f
I20260928 03:09:30.399894 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:30.388918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:43505 (local address 127.6.192.62:35561)
0928 03:09:30.389115 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:30.389120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:30.390598 (+ 1478us) server_negotiation.cc:407] Connection header received
0928 03:09:30.392746 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:30.392752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:30.392854 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:30.392972 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:30.394381 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.395223 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.396900 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.397328 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.397491 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:30.398956 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:30.398962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:30.398967 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:30.399003 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:30.399039 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:30.399051 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:30.399150 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:30.399389 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 03:09:30.399527 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 03:09:30.401024 26214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:30.389475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:35561 (local address 127.6.192.3:43505)
0928 03:09:30.390436 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:30.390479 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:30.392422 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:30.393309 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:30.393322 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:30.393467 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:30.394201 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:30.394219 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.395402 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.395412 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:30.396739 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:30.396751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.397597 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.397606 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:30.397847 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:30.398611 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:09:30.398614 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:30.398659 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:09:30.398742 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:30.399152 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:30.399166 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:30.399177 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 03:09:30.399376 (+ 199us) client_negotiation.cc:240] Negotiation successful
0928 03:09:30.399835 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":295,"threads_started":1}
I20260928 03:09:30.402516 26212 heartbeater.cc:347] Connected to a master server at 127.6.192.62:35561
I20260928 03:09:30.402995 26212 heartbeater.cc:464] Registering TS with master...
I20260928 03:09:30.403986 26212 heartbeater.cc:511] Master 127.6.192.62:35561 requested a full tablet report, sending...
I20260928 03:09:30.405742 25765 ts_manager.cc:194] Registered new tserver with Master: 2d2e3d66d39d4414885ff03755e20f3a (127.6.192.3:46519)
I20260928 03:09:30.406821 25765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.192.3:43505
I20260928 03:09:30.408203 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:09:30.420106 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:30.412647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50422 (local address 127.6.192.62:35561)
0928 03:09:30.412823 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:30.412828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:30.413343 (+ 515us) server_negotiation.cc:407] Connection header received
0928 03:09:30.413669 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:30.413673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:30.413751 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:30.413853 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:30.414456 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.415368 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.416837 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.417334 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.417435 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:30.419163 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:30.419170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:30.419174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:30.419201 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:30.419246 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:30.419263 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:30.419501 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:30.419711 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 03:09:30.419779 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 03:09:30.425241 25765 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:09:30.432358 25765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:09:30.449366 25951 heartbeater.cc:503] Master 127.6.192.62:35561 was elected leader, sending a full tablet report...
I20260928 03:09:30.472886 26228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:30.447032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44300 (local address 127.6.192.2:45367)
0928 03:09:30.461074 (+ 14042us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:30.461089 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:30.462135 (+ 1046us) server_negotiation.cc:407] Connection header received
0928 03:09:30.462254 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:30.462263 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:30.462645 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:30.462868 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:30.464245 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.466061 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.470123 (+ 4062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.471401 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.471624 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:30.471940 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:30.472282 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 03:09:30.472483 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13838,"thread_start_us":13696,"threads_started":1}
I20260928 03:09:30.473022 26227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:30.446265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:45367 (local address 127.0.0.1:44300)
0928 03:09:30.461777 (+ 15512us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:30.461801 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:30.461988 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:30.463227 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:30.463241 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:30.463288 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:30.463951 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:30.463970 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.466967 (+ 2997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.466973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:30.469850 (+ 2877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:30.469867 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.471689 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.471697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:30.471946 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:30.472022 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 03:09:30.472187 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 03:09:30.472525 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15298,"mutex_wait_us":1,"thread_start_us":20161,"threads_started":1}
I20260928 03:09:30.476130 26229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:30.447258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.6.192.1:41857)
0928 03:09:30.465770 (+ 18512us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:30.465784 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:30.466807 (+ 1023us) server_negotiation.cc:407] Connection header received
0928 03:09:30.466980 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:30.466990 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:30.467305 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:30.467633 (+ 328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:30.468696 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.470447 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.473134 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.474969 (+ 1835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.475111 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:30.475426 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:30.475621 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 03:09:30.475799 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18253,"thread_start_us":18535,"threads_started":1}
I20260928 03:09:30.476269 26225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:30.446581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41857 (local address 127.0.0.1:56866)
0928 03:09:30.466633 (+ 20052us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:30.466662 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:30.466825 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:30.467744 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:30.467750 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:30.467780 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:30.468265 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:30.468280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.470595 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.470601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:30.472832 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:30.472851 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.475161 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.475168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:30.475365 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:30.475413 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 03:09:30.475547 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 03:09:30.475698 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19917,"mutex_wait_us":39,"thread_start_us":14790,"threads_started":1}
I20260928 03:09:30.477828 26224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:30.446005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:46519 (local address 127.0.0.1:49408)
0928 03:09:30.459918 (+ 13913us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:30.459976 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:30.460260 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:30.469197 (+ 8937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:30.469201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:30.469230 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:30.469703 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:30.469716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.472134 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.472143 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:30.474603 (+ 2460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:30.474617 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.477013 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.477020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:30.477209 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:30.477251 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 03:09:30.477352 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 03:09:30.477496 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13478,"thread_start_us":12376,"threads_started":1}
I20260928 03:09:30.478006 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:30.446283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49408 (local address 127.6.192.3:46519)
0928 03:09:30.468548 (+ 22265us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:30.468555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:30.468585 (+ 30us) server_negotiation.cc:407] Connection header received
0928 03:09:30.468662 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:30.468667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:30.468921 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:30.469971 (+ 1050us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:30.470287 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.471978 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.475069 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.476796 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.476956 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:30.477257 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:30.477412 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 03:09:30.477580 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22086,"thread_start_us":21931,"threads_started":1}
I20260928 03:09:30.477782 26017 tablet_service.cc:1467] Processing CreateTablet for tablet 169ab16fb0354b2faf36135736b7ef6d (DEFAULT_TABLE table=test-table [id=88a6089f0a1a43afbe21833f3ff83602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:09:30.480232 25886 tablet_service.cc:1467] Processing CreateTablet for tablet 169ab16fb0354b2faf36135736b7ef6d (DEFAULT_TABLE table=test-table [id=88a6089f0a1a43afbe21833f3ff83602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:09:30.481812 25886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169ab16fb0354b2faf36135736b7ef6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:30.481824 26147 tablet_service.cc:1467] Processing CreateTablet for tablet 169ab16fb0354b2faf36135736b7ef6d (DEFAULT_TABLE table=test-table [id=88a6089f0a1a43afbe21833f3ff83602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:09:30.483300 26147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169ab16fb0354b2faf36135736b7ef6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:30.486599 26017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169ab16fb0354b2faf36135736b7ef6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:30.497984 26230 tablet_bootstrap.cc:492] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554: Bootstrap starting.
I20260928 03:09:30.500267 26231 tablet_bootstrap.cc:492] T 169ab16fb0354b2faf36135736b7ef6d P 5e758be175c046a8a48953fd2f59069d: Bootstrap starting.
I20260928 03:09:30.502091 26230 tablet_bootstrap.cc:654] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:30.503798 26230 log.cc:824] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:30.504139 26231 tablet_bootstrap.cc:654] T 169ab16fb0354b2faf36135736b7ef6d P 5e758be175c046a8a48953fd2f59069d: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:30.504190 26232 tablet_bootstrap.cc:492] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a: Bootstrap starting.
I20260928 03:09:30.505424 26231 log.cc:824] T 169ab16fb0354b2faf36135736b7ef6d P 5e758be175c046a8a48953fd2f59069d: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:30.507845 26230 tablet_bootstrap.cc:492] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554: No bootstrap required, opened a new log
I20260928 03:09:30.508167 26230 ts_tablet_manager.cc:1397] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 03:09:30.508652 26231 tablet_bootstrap.cc:492] T 169ab16fb0354b2faf36135736b7ef6d P 5e758be175c046a8a48953fd2f59069d: No bootstrap required, opened a new log
I20260928 03:09:30.508656 26232 tablet_bootstrap.cc:654] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:30.508934 26231 ts_tablet_manager.cc:1397] T 169ab16fb0354b2faf36135736b7ef6d P 5e758be175c046a8a48953fd2f59069d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260928 03:09:30.509862 26232 log.cc:824] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:30.512781 26232 tablet_bootstrap.cc:492] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a: No bootstrap required, opened a new log
I20260928 03:09:30.513070 26232 ts_tablet_manager.cc:1397] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260928 03:09:30.519928 26230 raft_consensus.cc:348] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2e3d66d39d4414885ff03755e20f3a" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46519 } } peers { permanent_uuid: "7e6d242e1dc746a6ad352868dc412554" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45367 } } peers { permanent_uuid: "5e758be175c046a8a48953fd2f59069d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41857 } }
I20260928 03:09:30.519928 26231 raft_consensus.cc:348] T 169ab16fb0354b2faf36135736b7ef6d P 5e758be175c046a8a48953fd2f59069d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2e3d66d39d4414885ff03755e20f3a" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46519 } } peers { permanent_uuid: "7e6d242e1dc746a6ad352868dc412554" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45367 } } peers { permanent_uuid: "5e758be175c046a8a48953fd2f59069d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41857 } }
I20260928 03:09:30.520453 26231 raft_consensus.cc:374] T 169ab16fb0354b2faf36135736b7ef6d P 5e758be175c046a8a48953fd2f59069d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:30.520453 26230 raft_consensus.cc:374] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:30.520565 26230 raft_consensus.cc:729] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e6d242e1dc746a6ad352868dc412554, State: Initialized, Role: FOLLOWER
I20260928 03:09:30.520593 26231 raft_consensus.cc:729] T 169ab16fb0354b2faf36135736b7ef6d P 5e758be175c046a8a48953fd2f59069d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e758be175c046a8a48953fd2f59069d, State: Initialized, Role: FOLLOWER
I20260928 03:09:30.521299 26231 consensus_queue.cc:260] T 169ab16fb0354b2faf36135736b7ef6d P 5e758be175c046a8a48953fd2f59069d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2e3d66d39d4414885ff03755e20f3a" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46519 } } peers { permanent_uuid: "7e6d242e1dc746a6ad352868dc412554" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45367 } } peers { permanent_uuid: "5e758be175c046a8a48953fd2f59069d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41857 } }
I20260928 03:09:30.521288 26230 consensus_queue.cc:260] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2e3d66d39d4414885ff03755e20f3a" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46519 } } peers { permanent_uuid: "7e6d242e1dc746a6ad352868dc412554" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45367 } } peers { permanent_uuid: "5e758be175c046a8a48953fd2f59069d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41857 } }
I20260928 03:09:30.523136 26082 heartbeater.cc:503] Master 127.6.192.62:35561 was elected leader, sending a full tablet report...
I20260928 03:09:30.523778 26230 ts_tablet_manager.cc:1428] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260928 03:09:30.524024 26231 ts_tablet_manager.cc:1428] T 169ab16fb0354b2faf36135736b7ef6d P 5e758be175c046a8a48953fd2f59069d: Time spent starting tablet: real 0.015s user 0.003s sys 0.013s
I20260928 03:09:30.524446 26232 raft_consensus.cc:348] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2e3d66d39d4414885ff03755e20f3a" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46519 } } peers { permanent_uuid: "7e6d242e1dc746a6ad352868dc412554" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45367 } } peers { permanent_uuid: "5e758be175c046a8a48953fd2f59069d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41857 } }
I20260928 03:09:30.524988 26232 raft_consensus.cc:374] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:30.525113 26232 raft_consensus.cc:729] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2e3d66d39d4414885ff03755e20f3a, State: Initialized, Role: FOLLOWER
I20260928 03:09:30.525816 26232 consensus_queue.cc:260] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2e3d66d39d4414885ff03755e20f3a" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46519 } } peers { permanent_uuid: "7e6d242e1dc746a6ad352868dc412554" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45367 } } peers { permanent_uuid: "5e758be175c046a8a48953fd2f59069d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41857 } }
I20260928 03:09:30.526937 26212 heartbeater.cc:503] Master 127.6.192.62:35561 was elected leader, sending a full tablet report...
I20260928 03:09:30.527709 26232 ts_tablet_manager.cc:1428] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
W20260928 03:09:30.638000 26213 tablet.cc:2369] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:09:30.646526 26083 tablet.cc:2369] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:09:30.673058 25952 tablet.cc:2369] T 169ab16fb0354b2faf36135736b7ef6d P 5e758be175c046a8a48953fd2f59069d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:09:30.853252 26238 raft_consensus.cc:482] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:09:30.853610 26238 raft_consensus.cc:504] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2e3d66d39d4414885ff03755e20f3a" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46519 } } peers { permanent_uuid: "7e6d242e1dc746a6ad352868dc412554" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45367 } } peers { permanent_uuid: "5e758be175c046a8a48953fd2f59069d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41857 } }
I20260928 03:09:30.855525 26238 leader_election.cc:288] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e6d242e1dc746a6ad352868dc412554 (127.6.192.2:45367), 5e758be175c046a8a48953fd2f59069d (127.6.192.1:41857)
I20260928 03:09:30.864027 26228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:30.856120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:57241 (local address 127.6.192.2:45367)
0928 03:09:30.856361 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:30.856371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:30.856400 (+ 29us) server_negotiation.cc:407] Connection header received
0928 03:09:30.856488 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:30.856494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:30.856587 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:30.856699 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:30.857367 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.858452 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.861494 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.862880 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.863063 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:30.863202 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:30.863574 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 03:09:30.863724 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 03:09:30.864054 26229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:30.856007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:46257 (local address 127.6.192.1:41857)
0928 03:09:30.856223 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:30.856230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:30.856798 (+ 568us) server_negotiation.cc:407] Connection header received
0928 03:09:30.856953 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:30.856957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:30.857024 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:30.857096 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:30.857642 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.858676 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.861494 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.863000 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.863127 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:30.863241 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:30.863567 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 03:09:30.863724 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 03:09:30.864071 26214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:30.855877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:45367 (local address 127.6.192.3:57241)
0928 03:09:30.856075 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:30.856099 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:30.856233 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:30.856771 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:30.856777 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:30.856830 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:30.857040 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:30.857050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.858669 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.858675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:30.861240 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:30.861261 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.863076 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.863085 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:30.863314 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:30.863395 (+ 81us) client_negotiation.cc:769] Sending connection context
0928 03:09:30.863524 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 03:09:30.863724 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":202,"spinlock_wait_cycles":25856}
I20260928 03:09:30.864137 26239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:30.855877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41857 (local address 127.6.192.3:46257)
0928 03:09:30.856651 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:30.856693 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:30.856813 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:30.857155 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:30.857161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:30.857191 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:30.857484 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:30.857491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.858838 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.858843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:30.861243 (+ 2400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:30.861261 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.863163 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.863169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:30.863361 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:30.863414 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 03:09:30.863524 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 03:09:30.863724 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":322,"thread_start_us":292,"threads_started":1}
I20260928 03:09:30.865131 25906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169ab16fb0354b2faf36135736b7ef6d" candidate_uuid: "2d2e3d66d39d4414885ff03755e20f3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e758be175c046a8a48953fd2f59069d" is_pre_election: true
I20260928 03:09:30.865698 25906 raft_consensus.cc:2454] T 169ab16fb0354b2faf36135736b7ef6d P 5e758be175c046a8a48953fd2f59069d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d2e3d66d39d4414885ff03755e20f3a in term 0.
I20260928 03:09:30.865609 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169ab16fb0354b2faf36135736b7ef6d" candidate_uuid: "2d2e3d66d39d4414885ff03755e20f3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e6d242e1dc746a6ad352868dc412554" is_pre_election: true
I20260928 03:09:30.866130 26037 raft_consensus.cc:2454] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d2e3d66d39d4414885ff03755e20f3a in term 0.
I20260928 03:09:30.866489 26101 leader_election.cc:302] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d2e3d66d39d4414885ff03755e20f3a, 5e758be175c046a8a48953fd2f59069d; no voters:
I20260928 03:09:30.867070 26238 raft_consensus.cc:2781] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:09:30.867290 26238 raft_consensus.cc:482] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:09:30.867393 26238 raft_consensus.cc:3037] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:30.869840 26238 raft_consensus.cc:504] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2e3d66d39d4414885ff03755e20f3a" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46519 } } peers { permanent_uuid: "7e6d242e1dc746a6ad352868dc412554" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45367 } } peers { permanent_uuid: "5e758be175c046a8a48953fd2f59069d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41857 } }
I20260928 03:09:30.870560 26238 leader_election.cc:288] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [CANDIDATE]: Term 1 election: Requested vote from peers 7e6d242e1dc746a6ad352868dc412554 (127.6.192.2:45367), 5e758be175c046a8a48953fd2f59069d (127.6.192.1:41857)
I20260928 03:09:30.871239 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169ab16fb0354b2faf36135736b7ef6d" candidate_uuid: "2d2e3d66d39d4414885ff03755e20f3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e6d242e1dc746a6ad352868dc412554"
I20260928 03:09:30.871250 25906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169ab16fb0354b2faf36135736b7ef6d" candidate_uuid: "2d2e3d66d39d4414885ff03755e20f3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e758be175c046a8a48953fd2f59069d"
I20260928 03:09:30.871521 26037 raft_consensus.cc:3037] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:30.871582 25906 raft_consensus.cc:3037] T 169ab16fb0354b2faf36135736b7ef6d P 5e758be175c046a8a48953fd2f59069d [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:30.874200 25906 raft_consensus.cc:2454] T 169ab16fb0354b2faf36135736b7ef6d P 5e758be175c046a8a48953fd2f59069d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d2e3d66d39d4414885ff03755e20f3a in term 1.
I20260928 03:09:30.874208 26037 raft_consensus.cc:2454] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d2e3d66d39d4414885ff03755e20f3a in term 1.
I20260928 03:09:30.874938 26103 leader_election.cc:302] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2d2e3d66d39d4414885ff03755e20f3a, 5e758be175c046a8a48953fd2f59069d, 7e6d242e1dc746a6ad352868dc412554; no voters:
I20260928 03:09:30.875381 26238 raft_consensus.cc:2781] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:30.875947 26238 raft_consensus.cc:686] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [term 1 LEADER]: Becoming Leader. State: Replica: 2d2e3d66d39d4414885ff03755e20f3a, State: Running, Role: LEADER
I20260928 03:09:30.876471 26238 consensus_queue.cc:237] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2e3d66d39d4414885ff03755e20f3a" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46519 } } peers { permanent_uuid: "7e6d242e1dc746a6ad352868dc412554" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45367 } } peers { permanent_uuid: "5e758be175c046a8a48953fd2f59069d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41857 } }
I20260928 03:09:30.881511 25764 catalog_manager.cc:5495] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d2e3d66d39d4414885ff03755e20f3a (127.6.192.3). New cstate: current_term: 1 leader_uuid: "2d2e3d66d39d4414885ff03755e20f3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2e3d66d39d4414885ff03755e20f3a" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e6d242e1dc746a6ad352868dc412554" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e758be175c046a8a48953fd2f59069d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41857 } health_report { overall_health: UNKNOWN } } }
I20260928 03:09:30.935966 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:30.929227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49424 (local address 127.6.192.3:46519)
0928 03:09:30.929456 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:30.929461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:30.929963 (+ 502us) server_negotiation.cc:407] Connection header received
0928 03:09:30.930146 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:30.930150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:30.930250 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:30.930422 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:09:30.930996 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.931882 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.933455 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:30.933926 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:30.934034 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:30.934557 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:09:30.935181 (+ 624us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:09:30.935318 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:30.935535 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 03:09:30.935601 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 03:09:30.945257 25906 raft_consensus.cc:1261] T 169ab16fb0354b2faf36135736b7ef6d P 5e758be175c046a8a48953fd2f59069d [term 1 FOLLOWER]: Refusing update from remote peer 2d2e3d66d39d4414885ff03755e20f3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:09:30.945257 26037 raft_consensus.cc:1261] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554 [term 1 FOLLOWER]: Refusing update from remote peer 2d2e3d66d39d4414885ff03755e20f3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:09:30.946568 26238 consensus_queue.cc:1035] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d242e1dc746a6ad352868dc412554" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:09:30.946990 26240 consensus_queue.cc:1035] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e758be175c046a8a48953fd2f59069d" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 41857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:09:30.983387 25764 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:09:30.983789 25764 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:09:30.987368 25764 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d8834e0bc34441385bb0622d868a487: Sending DeleteTablet for 3 replicas of tablet 169ab16fb0354b2faf36135736b7ef6d
I20260928 03:09:30.988673 26017 tablet_service.cc:1514] Processing DeleteTablet for tablet 169ab16fb0354b2faf36135736b7ef6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:09:30 UTC) from {username='slave'} at 127.0.0.1:44300
I20260928 03:09:30.988680 26147 tablet_service.cc:1514] Processing DeleteTablet for tablet 169ab16fb0354b2faf36135736b7ef6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:09:30 UTC) from {username='slave'} at 127.0.0.1:49408
I20260928 03:09:30.989187 25886 tablet_service.cc:1514] Processing DeleteTablet for tablet 169ab16fb0354b2faf36135736b7ef6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:09:30 UTC) from {username='slave'} at 127.0.0.1:56866
I20260928 03:09:30.991279 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 25824
I20260928 03:09:31.000154 26255 tablet_replica.cc:330] T 169ab16fb0354b2faf36135736b7ef6d P 5e758be175c046a8a48953fd2f59069d: stopping tablet replica
I20260928 03:09:31.000785 26255 raft_consensus.cc:2229] T 169ab16fb0354b2faf36135736b7ef6d P 5e758be175c046a8a48953fd2f59069d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:09:31.001489 26255 raft_consensus.cc:2258] T 169ab16fb0354b2faf36135736b7ef6d P 5e758be175c046a8a48953fd2f59069d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:09:31.005038 26254 tablet_replica.cc:330] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a: stopping tablet replica
I20260928 03:09:31.005995 26254 raft_consensus.cc:2229] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:09:31.007017 26254 raft_consensus.cc:2258] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:09:31.010141 26254 ts_tablet_manager.cc:1905] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:09:31.015317 26255 ts_tablet_manager.cc:1905] T 169ab16fb0354b2faf36135736b7ef6d P 5e758be175c046a8a48953fd2f59069d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:09:31.016808 26253 tablet_replica.cc:330] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554: stopping tablet replica
I20260928 03:09:31.017305 26253 raft_consensus.cc:2229] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:09:31.019192 26254 ts_tablet_manager.cc:1918] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:09:31.019397 26254 log.cc:1196] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/169ab16fb0354b2faf36135736b7ef6d
I20260928 03:09:31.019984 26229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:30.992764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56876 (local address 127.6.192.1:41857)
0928 03:09:30.992972 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:30.992979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:31.006965 (+ 13986us) server_negotiation.cc:407] Connection header received
0928 03:09:31.007554 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:31.007561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:31.007686 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:31.007787 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:31.009386 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:31.010532 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:31.012701 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:31.013406 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:31.013513 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:31.018011 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:31.018024 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:31.018042 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:31.018870 (+ 828us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:31.018942 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:31.018972 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:31.019092 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:31.019381 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 03:09:31.019560 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 03:09:31.020027 26254 ts_tablet_manager.cc:1939] T 169ab16fb0354b2faf36135736b7ef6d P 2d2e3d66d39d4414885ff03755e20f3a: Deleting consensus metadata
I20260928 03:09:31.021580 25946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:31.022112 25749 catalog_manager.cc:4841] TS 2d2e3d66d39d4414885ff03755e20f3a (127.6.192.3:46519): tablet 169ab16fb0354b2faf36135736b7ef6d (table test-table [id=88a6089f0a1a43afbe21833f3ff83602]) successfully deleted
I20260928 03:09:31.022462 26253 raft_consensus.cc:2258] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:09:31.024417 26253 ts_tablet_manager.cc:1905] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:09:31.032037 26253 ts_tablet_manager.cc:1918] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:09:31.032261 26253 log.cc:1196] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/169ab16fb0354b2faf36135736b7ef6d
I20260928 03:09:31.032794 26253 ts_tablet_manager.cc:1939] T 169ab16fb0354b2faf36135736b7ef6d P 7e6d242e1dc746a6ad352868dc412554: Deleting consensus metadata
I20260928 03:09:31.034881 25752 catalog_manager.cc:4841] TS 7e6d242e1dc746a6ad352868dc412554 (127.6.192.2:45367): tablet 169ab16fb0354b2faf36135736b7ef6d (table test-table [id=88a6089f0a1a43afbe21833f3ff83602]) successfully deleted
I20260928 03:09:31.124197 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 25824
W20260928 03:09:31.142040 25750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.192.1:41857 (error 108)
I20260928 03:09:31.142804 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 25955
I20260928 03:09:31.144577 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:31.143410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41857 (local address 127.0.0.1:56878)
0928 03:09:31.144192 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:31.144354 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":472,"threads_started":1}
W20260928 03:09:31.144887 25750 catalog_manager.cc:4584] TS 5e758be175c046a8a48953fd2f59069d (127.6.192.1:41857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 169ab16fb0354b2faf36135736b7ef6d: Network error: Client connection negotiation failed: client connection to 127.6.192.1:41857: connect: Connection refused (error 111)
I20260928 03:09:31.151259 26228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:31.143691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44304 (local address 127.6.192.2:45367)
0928 03:09:31.143904 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:31.143910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:31.144192 (+ 282us) server_negotiation.cc:407] Connection header received
0928 03:09:31.144380 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:31.144386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:31.144493 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:31.144626 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:31.145212 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:31.145949 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:31.147905 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:31.148406 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:31.148503 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:31.149718 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:31.149726 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:31.149740 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:31.150385 (+ 645us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:31.150435 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:31.150455 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:31.150549 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:31.150703 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 03:09:31.150827 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 03:09:31.152114 26077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:31.198851 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:31.198244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41857 (local address 127.0.0.1:56882)
0928 03:09:31.198548 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:31.198673 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 03:09:31.200206 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:31.199710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41857 (local address 127.0.0.1:56890)
0928 03:09:31.199926 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:31.200023 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 03:09:31.244518 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 25955
I20260928 03:09:31.261090 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:31.260565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41857 (local address 127.0.0.1:48420)
0928 03:09:31.260789 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:31.260905 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 03:09:31.262454 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:31.261955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41857 (local address 127.0.0.1:48424)
0928 03:09:31.262107 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:31.262275 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 03:09:31.268556 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 26085
I20260928 03:09:31.276120 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:31.269457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.6.192.3:46519)
0928 03:09:31.269623 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:31.269628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:31.269647 (+ 19us) server_negotiation.cc:407] Connection header received
0928 03:09:31.269821 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:31.269826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:31.269912 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:31.270006 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:31.270483 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:31.271380 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:31.272831 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:31.273286 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:31.273431 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:31.274553 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:31.274562 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:31.274576 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:31.275212 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:31.275276 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:31.275312 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:31.275453 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:31.275650 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 03:09:31.275759 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 03:09:31.276857 26207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:31.343351 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:31.342643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41857 (local address 127.0.0.1:48434)
0928 03:09:31.342987 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:31.343141 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260928 03:09:31.344867 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:31.344301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41857 (local address 127.0.0.1:48444)
0928 03:09:31.344577 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:31.344663 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260928 03:09:31.402035 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 26085
I20260928 03:09:31.421995 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 25734
I20260928 03:09:31.423329 25795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:31.535364 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 25734
I20260928 03:09:31.535542 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:31.534886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:41857 (local address 127.0.0.1:48458)
0928 03:09:31.535172 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:31.535312 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.1:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
2026-09-28T03:09:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3291 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:09:31 dist-test-slave-fjww krb5kdc[26265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:09:31 dist-test-slave-fjww krb5kdc[26265](info): set up 2 sockets
Sep 28 03:09:31 dist-test-slave-fjww krb5kdc[26265](info): commencing operation
krb5kdc: starting...
W20260928 03:09:32.672194 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:09:32 dist-test-slave-fjww krb5kdc[26265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:09:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:09:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:09:32.948673 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:46095
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:37767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:46095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:33.291976 26278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:33.292418 26278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:33.292590 26278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:33.300375 26278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:09:33.300550 26278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:33.300659 26278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:33.300760 26278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:09:33.300846 26278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:09:33.311517 26278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:37767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:46095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:46095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:33.312991 26278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:33.314396 26278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:33.327348 26286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:33.327288 26284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:33.327260 26283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:33.328253 26278 server_base.cc:1034] running on GCE node
I20260928 03:09:33.329110 26278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:33.330651 26278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:33.331892 26278 hybrid_clock.cc:648] HybridClock initialized: now 1790564973331840 us; error 48 us; skew 500 ppm
I20260928 03:09:33.332288 26278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:33 dist-test-slave-fjww krb5kdc[26265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564973, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:33.335553 26278 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:09:33.337242 26278 webserver.cc:460] Webserver started at http://127.6.192.62:45683/ using document root <none> and password file <none>
I20260928 03:09:33.338008 26278 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:33.338165 26278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:33.338495 26278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:33.341722 26278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "997befca13d84a9390068d7c5dfcda0f"
format_stamp: "Formatted at 2026-09-28 03:09:33 on dist-test-slave-fjww"
server_key: "e1b249cda44a1c70bb68f5310b166ef2"
server_key_iv: "b47f3db6e8ba3cf8f8c98601d904965a"
server_key_version: "encryptionkey@0"
I20260928 03:09:33.342595 26278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "997befca13d84a9390068d7c5dfcda0f"
format_stamp: "Formatted at 2026-09-28 03:09:33 on dist-test-slave-fjww"
server_key: "e1b249cda44a1c70bb68f5310b166ef2"
server_key_iv: "b47f3db6e8ba3cf8f8c98601d904965a"
server_key_version: "encryptionkey@0"
I20260928 03:09:33.348431 26278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 03:09:33.352783 26293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:33.354708 26278 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:09:33.354951 26278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "997befca13d84a9390068d7c5dfcda0f"
format_stamp: "Formatted at 2026-09-28 03:09:33 on dist-test-slave-fjww"
server_key: "e1b249cda44a1c70bb68f5310b166ef2"
server_key_iv: "b47f3db6e8ba3cf8f8c98601d904965a"
server_key_version: "encryptionkey@0"
I20260928 03:09:33.355185 26278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:33.374101 26278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:33.378826 26278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:33.379153 26278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:33.399072 26278 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:46095
I20260928 03:09:33.399104 26344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:46095 every 8 connection(s)
I20260928 03:09:33.400599 26278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:09:33.404593 26345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:33.405539 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 26278
I20260928 03:09:33.405725 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:09:33.406288 6912 external_mini_cluster.cc:1428] Setting key cb9863e78e60365a9142df1b213c44d8
I20260928 03:09:33.419519 26345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f: Bootstrap starting.
Sep 28 03:09:33 dist-test-slave-fjww krb5kdc[26265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:09:33.424041 26345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:33.425285 26345 log.cc:824] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:33.429358 26345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f: No bootstrap required, opened a new log
I20260928 03:09:33.434031 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:33.408378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38020 (local address 127.6.192.62:46095)
0928 03:09:33.409223 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:33.409238 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:33.409282 (+ 44us) server_negotiation.cc:407] Connection header received
0928 03:09:33.411159 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:33.411190 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:33.411769 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:33.412153 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:33.414032 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:33.415332 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:33.417362 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:33.418142 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:33.418289 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:33.423961 (+ 5672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:33.423970 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:33.423987 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:33.424048 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:33.428549 (+ 4501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:33.429411 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:33.429425 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:33.429435 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:33.429690 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:33.430103 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:33.430109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:33.430112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:33.430984 (+ 872us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:33.431333 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:33.431775 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 03:09:33.432362 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":397,"threads_started":1}
I20260928 03:09:33.439721 26345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997befca13d84a9390068d7c5dfcda0f" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46095 } }
I20260928 03:09:33.440153 26345 raft_consensus.cc:374] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:33.440238 26345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997befca13d84a9390068d7c5dfcda0f, State: Initialized, Role: FOLLOWER
I20260928 03:09:33.440809 26345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997befca13d84a9390068d7c5dfcda0f" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46095 } }
I20260928 03:09:33.441033 26345 raft_consensus.cc:388] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:09:33.441144 26345 raft_consensus.cc:482] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:09:33.441291 26345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:33.444033 26345 raft_consensus.cc:504] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997befca13d84a9390068d7c5dfcda0f" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46095 } }
I20260928 03:09:33.444597 26345 leader_election.cc:302] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 997befca13d84a9390068d7c5dfcda0f; no voters:
I20260928 03:09:33.445238 26345 leader_election.cc:288] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:09:33.445564 26350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:33.446919 26350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f [term 1 LEADER]: Becoming Leader. State: Replica: 997befca13d84a9390068d7c5dfcda0f, State: Running, Role: LEADER
I20260928 03:09:33.447551 26350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997befca13d84a9390068d7c5dfcda0f" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46095 } }
I20260928 03:09:33.448040 26345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:09:33.451122 26352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 997befca13d84a9390068d7c5dfcda0f. Latest consensus state: current_term: 1 leader_uuid: "997befca13d84a9390068d7c5dfcda0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997befca13d84a9390068d7c5dfcda0f" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46095 } } }
I20260928 03:09:33.451186 26351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "997befca13d84a9390068d7c5dfcda0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997befca13d84a9390068d7c5dfcda0f" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46095 } } }
I20260928 03:09:33.451766 26352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:33.451812 26351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:33.453575 26359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:09:33.459244 26359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:09:33.468791 26359 catalog_manager.cc:1347] Generated new cluster ID: 985bb435750f4538ab90e5936d6055be
I20260928 03:09:33.468925 26359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:09:33.481597 26359 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:09:33.483021 26359 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:09:33.495039 26359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f: Generated new TSK 0
I20260928 03:09:33.496281 26359 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.192.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
Entry for principal kudu/127.6.192.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab.
I20260928 03:09:33.567358 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:46095
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:37767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:33.922453 26371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:33.922773 26371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:33.922881 26371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:33.931715 26371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:33.931859 26371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:33.932001 26371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:09:33.944432 26371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:37767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.1.keytab
--principal=kudu/127.6.192.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:46095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:33.945900 26371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:33.947775 26371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:33.962800 26379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:33.962801 26377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:33.962826 26376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:33.964124 26371 server_base.cc:1034] running on GCE node
I20260928 03:09:33.965013 26371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:33.966114 26371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:33.967379 26371 hybrid_clock.cc:648] HybridClock initialized: now 1790564973967300 us; error 69 us; skew 500 ppm
I20260928 03:09:33.967799 26371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:33 dist-test-slave-fjww krb5kdc[26265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564973, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:33.971200 26371 init.cc:377] Logged in from keytab as kudu/127.6.192.1@KRBTEST.COM (short username kudu)
I20260928 03:09:33.973007 26371 webserver.cc:460] Webserver started at http://127.6.192.1:41647/ using document root <none> and password file <none>
I20260928 03:09:33.973762 26371 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:33.973913 26371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:33.974246 26371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:33.977455 26371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "eab3d5e6d3cf4b0f95391cbaae71fefe"
format_stamp: "Formatted at 2026-09-28 03:09:33 on dist-test-slave-fjww"
server_key: "48c9c6bdc10d0118c169247ba48dc73e"
server_key_iv: "18ff520007898fe1122c6b29872d546d"
server_key_version: "encryptionkey@0"
I20260928 03:09:33.978399 26371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "eab3d5e6d3cf4b0f95391cbaae71fefe"
format_stamp: "Formatted at 2026-09-28 03:09:33 on dist-test-slave-fjww"
server_key: "48c9c6bdc10d0118c169247ba48dc73e"
server_key_iv: "18ff520007898fe1122c6b29872d546d"
server_key_version: "encryptionkey@0"
I20260928 03:09:33.984313 26371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 03:09:33.988550 26386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:33.990216 26371 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 03:09:33.990449 26371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "eab3d5e6d3cf4b0f95391cbaae71fefe"
format_stamp: "Formatted at 2026-09-28 03:09:33 on dist-test-slave-fjww"
server_key: "48c9c6bdc10d0118c169247ba48dc73e"
server_key_iv: "18ff520007898fe1122c6b29872d546d"
server_key_version: "encryptionkey@0"
I20260928 03:09:33.990705 26371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:34.020675 26371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:34.025694 26371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:34.026018 26371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:34.027129 26371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:34.028676 26371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:34.028785 26371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:34.028946 26371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:34.029021 26371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:34.065140 26371 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:37469
I20260928 03:09:34.065181 26499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:37469 every 8 connection(s)
I20260928 03:09:34.066644 26371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:09:34.075124 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 26371
I20260928 03:09:34.075336 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:09:34.075845 6912 external_mini_cluster.cc:1428] Setting key 62e3ec97eb272b32eb430e518ea7ed14
Sep 28 03:09:34 dist-test-slave-fjww krb5kdc[26265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564973, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.1@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:09:34.090253 26502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:34.069552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:45731 (local address 127.6.192.62:46095)
0928 03:09:34.070029 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:34.070036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:34.071131 (+ 1095us) server_negotiation.cc:407] Connection header received
0928 03:09:34.073111 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:34.073116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:34.073192 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:34.073296 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:34.075069 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:34.076081 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:34.078097 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:34.078524 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:34.078623 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:34.083303 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:34.083309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:34.083312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:34.083354 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:34.086177 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:34.087335 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:34.087340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:34.087342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:34.087548 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:34.088156 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:34.088164 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:34.088170 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:34.088577 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:34.088729 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:34.089460 (+ 731us) server_negotiation.cc:299] Negotiation successful
0928 03:09:34.089737 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":222,"threads_started":1}
I20260928 03:09:34.090953 26503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:34.070187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:46095 (local address 127.6.192.1:45731)
0928 03:09:34.070955 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:34.070995 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:34.072819 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:34.073549 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:34.073559 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:34.074113 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:34.074847 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:34.074879 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:34.076249 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:34.076254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:34.077955 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:34.077972 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:34.078784 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:34.078792 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:34.079161 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:34.080005 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:09:34.080043 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:34.083012 (+ 2969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:34.086372 (+ 3360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:34.086382 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:34.086410 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:34.087166 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:34.087677 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:34.087682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:34.087685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:34.088016 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:34.088712 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:34.088718 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:34.089071 (+ 353us) client_negotiation.cc:769] Sending connection context
0928 03:09:34.089313 (+ 242us) client_negotiation.cc:240] Negotiation successful
0928 03:09:34.089747 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":314,"threads_started":1}
I20260928 03:09:34.092665 26500 heartbeater.cc:347] Connected to a master server at 127.6.192.62:46095
I20260928 03:09:34.093014 26500 heartbeater.cc:464] Registering TS with master...
I20260928 03:09:34.093775 26500 heartbeater.cc:511] Master 127.6.192.62:46095 requested a full tablet report, sending...
I20260928 03:09:34.096266 26310 ts_manager.cc:194] Registered new tserver with Master: eab3d5e6d3cf4b0f95391cbaae71fefe (127.6.192.1:37469)
I20260928 03:09:34.097756 26310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.1@KRBTEST.COM'} at 127.6.192.1:45731
WARNING: no policy specified for kudu/127.6.192.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
Entry for principal kudu/127.6.192.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab.
I20260928 03:09:34.137084 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:46095
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:37767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:34.481083 26506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:34.481463 26506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:34.481596 26506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:34.489387 26506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:34.489514 26506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:34.489630 26506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:09:34.500865 26506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:37767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.2.keytab
--principal=kudu/127.6.192.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:46095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:34.502298 26506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:34.504146 26506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:34.518508 26511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:34.518575 26512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:34.518555 26514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:34.520025 26506 server_base.cc:1034] running on GCE node
I20260928 03:09:34.520841 26506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:34.521974 26506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:34.523247 26506 hybrid_clock.cc:648] HybridClock initialized: now 1790564974523206 us; error 44 us; skew 500 ppm
I20260928 03:09:34.523708 26506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:34 dist-test-slave-fjww krb5kdc[26265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564974, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:34.527684 26506 init.cc:377] Logged in from keytab as kudu/127.6.192.2@KRBTEST.COM (short username kudu)
I20260928 03:09:34.529444 26506 webserver.cc:460] Webserver started at http://127.6.192.2:36699/ using document root <none> and password file <none>
I20260928 03:09:34.530222 26506 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:34.530336 26506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:34.530632 26506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:34.534085 26506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "9965bd6bccd14e4399a5ce8bd18584dd"
format_stamp: "Formatted at 2026-09-28 03:09:34 on dist-test-slave-fjww"
server_key: "a36d326d5136104ea29a1dd82c4c213d"
server_key_iv: "79461b6f54a4b699f502bb4cc23ad235"
server_key_version: "encryptionkey@0"
I20260928 03:09:34.534969 26506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "9965bd6bccd14e4399a5ce8bd18584dd"
format_stamp: "Formatted at 2026-09-28 03:09:34 on dist-test-slave-fjww"
server_key: "a36d326d5136104ea29a1dd82c4c213d"
server_key_iv: "79461b6f54a4b699f502bb4cc23ad235"
server_key_version: "encryptionkey@0"
I20260928 03:09:34.541158 26506 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 03:09:34.545075 26521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:34.546602 26506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:09:34.546757 26506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "9965bd6bccd14e4399a5ce8bd18584dd"
format_stamp: "Formatted at 2026-09-28 03:09:34 on dist-test-slave-fjww"
server_key: "a36d326d5136104ea29a1dd82c4c213d"
server_key_iv: "79461b6f54a4b699f502bb4cc23ad235"
server_key_version: "encryptionkey@0"
I20260928 03:09:34.547060 26506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:34.563598 26506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:34.568063 26506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:34.568316 26506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:34.569392 26506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:34.570829 26506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:34.570938 26506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:34.571095 26506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:34.571172 26506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:34.607347 26506 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:35449
I20260928 03:09:34.607405 26633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:35449 every 8 connection(s)
I20260928 03:09:34.608729 26506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:09:34.614239 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 26506
I20260928 03:09:34.614481 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:09:34.614961 6912 external_mini_cluster.cc:1428] Setting key 894718477b1c3a6488b037f206660b17
Sep 28 03:09:34 dist-test-slave-fjww krb5kdc[26265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564974, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.2@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:09:34.631140 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:34.611791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:60699 (local address 127.6.192.62:46095)
0928 03:09:34.612291 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:34.612296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:34.613523 (+ 1227us) server_negotiation.cc:407] Connection header received
0928 03:09:34.615712 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:34.615717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:34.615825 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:34.615950 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:34.617536 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:34.618363 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:34.620068 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:34.620514 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:34.620655 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:34.624894 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:34.624899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:34.624903 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:34.624941 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:34.627964 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:34.628758 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:34.628763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:34.628765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:34.628894 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:34.629349 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:34.629359 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:34.629361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:34.629652 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:34.629826 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:34.630533 (+ 707us) server_negotiation.cc:299] Negotiation successful
0928 03:09:34.630763 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":263,"threads_started":1}
I20260928 03:09:34.632229 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:34.612508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:46095 (local address 127.6.192.2:60699)
0928 03:09:34.613336 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:34.613386 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:34.615308 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:34.616212 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:34.616221 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:34.616763 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:34.617353 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:34.617371 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:34.618540 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:34.618544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:34.619912 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:34.619926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:34.620767 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:34.620775 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:34.621032 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:34.621838 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:09:34.621863 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:34.624594 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:34.628122 (+ 3528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:34.628131 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:34.628149 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:34.628611 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:34.629018 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:34.629023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:34.629027 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:34.629227 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:34.629787 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:34.629797 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:34.630166 (+ 369us) client_negotiation.cc:769] Sending connection context
0928 03:09:34.630444 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 03:09:34.631004 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":305,"threads_started":1}
I20260928 03:09:34.633926 26634 heartbeater.cc:347] Connected to a master server at 127.6.192.62:46095
I20260928 03:09:34.634289 26634 heartbeater.cc:464] Registering TS with master...
I20260928 03:09:34.635247 26634 heartbeater.cc:511] Master 127.6.192.62:46095 requested a full tablet report, sending...
I20260928 03:09:34.637091 26310 ts_manager.cc:194] Registered new tserver with Master: 9965bd6bccd14e4399a5ce8bd18584dd (127.6.192.2:35449)
I20260928 03:09:34.638648 26310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.2@KRBTEST.COM'} at 127.6.192.2:60699
WARNING: no policy specified for kudu/127.6.192.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
Entry for principal kudu/127.6.192.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab.
I20260928 03:09:34.679528 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:46095
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:37767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:35.044836 26640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:35.045145 26640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:35.045306 26640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:35.053154 26640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:35.053301 26640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:35.053496 26640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:09:35.064061 26640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:37767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.3.keytab
--principal=kudu/127.6.192.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:46095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:35.065366 26640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:35.066661 26640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:35.080335 26645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:35.080425 26646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:35.080678 26648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:35.081714 26640 server_base.cc:1034] running on GCE node
I20260928 03:09:35.082382 26640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:35.083400 26640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:35.084720 26640 hybrid_clock.cc:648] HybridClock initialized: now 1790564975084629 us; error 51 us; skew 500 ppm
I20260928 03:09:35.085191 26640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:35 dist-test-slave-fjww krb5kdc[26265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564975, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:35.088702 26640 init.cc:377] Logged in from keytab as kudu/127.6.192.3@KRBTEST.COM (short username kudu)
I20260928 03:09:35.090372 26640 webserver.cc:460] Webserver started at http://127.6.192.3:39391/ using document root <none> and password file <none>
I20260928 03:09:35.091210 26640 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:35.091360 26640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:35.091729 26640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:35.094936 26640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "72c81d5556a44e69b5c52c75c3bf6ec3"
format_stamp: "Formatted at 2026-09-28 03:09:35 on dist-test-slave-fjww"
server_key: "91af0b1fca45587543a17b17f83b23ee"
server_key_iv: "252d5005e1fedb4e7c58e1e6c3e0d5ca"
server_key_version: "encryptionkey@0"
I20260928 03:09:35.095963 26640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "72c81d5556a44e69b5c52c75c3bf6ec3"
format_stamp: "Formatted at 2026-09-28 03:09:35 on dist-test-slave-fjww"
server_key: "91af0b1fca45587543a17b17f83b23ee"
server_key_iv: "252d5005e1fedb4e7c58e1e6c3e0d5ca"
server_key_version: "encryptionkey@0"
I20260928 03:09:35.101023 26500 heartbeater.cc:503] Master 127.6.192.62:46095 was elected leader, sending a full tablet report...
I20260928 03:09:35.101822 26640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:09:35.106232 26655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:35.107919 26640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:09:35.108124 26640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "72c81d5556a44e69b5c52c75c3bf6ec3"
format_stamp: "Formatted at 2026-09-28 03:09:35 on dist-test-slave-fjww"
server_key: "91af0b1fca45587543a17b17f83b23ee"
server_key_iv: "252d5005e1fedb4e7c58e1e6c3e0d5ca"
server_key_version: "encryptionkey@0"
I20260928 03:09:35.108350 26640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:35.129953 26640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:35.134836 26640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:35.135107 26640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:35.136262 26640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:35.137624 26640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:35.137703 26640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:35.137787 26640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:35.137822 26640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:35.172649 26640 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:46509
I20260928 03:09:35.172729 26767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:46509 every 8 connection(s)
I20260928 03:09:35.174023 26640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:09:35.177165 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 26640
I20260928 03:09:35.177363 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:09:35.177919 6912 external_mini_cluster.cc:1428] Setting key bb852135e06f725f698b513dd21109c4
Sep 28 03:09:35 dist-test-slave-fjww krb5kdc[26265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564975, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.3@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:09:35.197489 26770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:35.177070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:41865 (local address 127.6.192.62:46095)
0928 03:09:35.177756 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:35.177763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:35.178799 (+ 1036us) server_negotiation.cc:407] Connection header received
0928 03:09:35.180915 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:35.180919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:35.180989 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:35.181076 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:35.183039 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.183910 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.185656 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.186159 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.186278 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:35.190288 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:35.190293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:35.190297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:35.190333 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:35.193793 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:35.194866 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:35.194873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:35.194878 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:35.195054 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:35.195602 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:35.195606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:35.195610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:35.195947 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:35.196122 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:35.196847 (+ 725us) server_negotiation.cc:299] Negotiation successful
0928 03:09:35.197089 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":393,"threads_started":1}
I20260928 03:09:35.198454 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:35.177628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:46095 (local address 127.6.192.3:41865)
0928 03:09:35.178589 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:35.178649 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:35.180641 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:35.181444 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:35.181455 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:35.182157 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:35.182827 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:35.182854 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.184060 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.184065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:35.185482 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:35.185496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.186389 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.186394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:35.186609 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:35.187404 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:09:35.187430 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:35.190060 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:35.193924 (+ 3864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:35.193932 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:35.193946 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:35.194712 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:35.195201 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:09:35.195206 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:09:35.195210 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:09:35.195419 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:09:35.196083 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:35.196093 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:35.196481 (+ 388us) client_negotiation.cc:769] Sending connection context
0928 03:09:35.196753 (+ 272us) client_negotiation.cc:240] Negotiation successful
0928 03:09:35.197310 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":428,"threads_started":1}
I20260928 03:09:35.199927 26768 heartbeater.cc:347] Connected to a master server at 127.6.192.62:46095
I20260928 03:09:35.200255 26768 heartbeater.cc:464] Registering TS with master...
I20260928 03:09:35.201045 26768 heartbeater.cc:511] Master 127.6.192.62:46095 requested a full tablet report, sending...
I20260928 03:09:35.202740 26310 ts_manager.cc:194] Registered new tserver with Master: 72c81d5556a44e69b5c52c75c3bf6ec3 (127.6.192.3:46509)
I20260928 03:09:35.203678 26310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.192.3@KRBTEST.COM'} at 127.6.192.3:41865
I20260928 03:09:35.206110 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:09:35.223380 26770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:35.210373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38026 (local address 127.6.192.62:46095)
0928 03:09:35.210541 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:35.210545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:35.210976 (+ 431us) server_negotiation.cc:407] Connection header received
0928 03:09:35.211119 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:35.211122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:35.211184 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:35.211267 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:35.212995 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.213715 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.215040 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.215526 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.215624 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:35.217991 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:35.217996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:35.218000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:35.218035 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:35.220789 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:35.221492 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:35.221496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:35.221498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:35.221622 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:35.221950 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:35.221953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:35.221955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:35.222274 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:35.222455 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:35.222938 (+ 483us) server_negotiation.cc:299] Negotiation successful
0928 03:09:35.223129 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 03:09:35.227705 26310 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:09:35.234611 26310 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:09:35.258390 26782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:35.248351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58460 (local address 127.6.192.3:46509)
0928 03:09:35.249095 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:35.249104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:35.249295 (+ 191us) server_negotiation.cc:407] Connection header received
0928 03:09:35.249658 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:35.249670 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:35.249954 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:35.250137 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:35.251148 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.252835 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.255622 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.257168 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.257306 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:35.257573 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:35.257880 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 03:09:35.258062 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":364,"threads_started":1}
I20260928 03:09:35.258442 26781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:35.248138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:46509 (local address 127.0.0.1:58460)
0928 03:09:35.249102 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:35.249150 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:35.249367 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:35.250477 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:35.250485 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:35.250513 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:35.250929 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:35.250949 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.253046 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.253051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:35.255398 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:35.255412 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.257382 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.257389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:35.257610 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:35.257667 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 03:09:35.257811 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 03:09:35.258009 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":511,"threads_started":1}
I20260928 03:09:35.259701 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:35.250040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:35449 (local address 127.0.0.1:35338)
0928 03:09:35.250554 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:35.250574 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:35.250690 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:35.252006 (+ 1316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:35.252011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:35.252037 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:35.252422 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:35.252434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.254667 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.254673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:35.256771 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:35.256783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.258790 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.258798 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:35.259007 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:35.259057 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 03:09:35.259141 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 03:09:35.259287 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":4,"thread_start_us":2376,"threads_started":1}
I20260928 03:09:35.261044 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:35.250686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35338 (local address 127.6.192.2:35449)
0928 03:09:35.251323 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:35.251332 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:35.251370 (+ 38us) server_negotiation.cc:407] Connection header received
0928 03:09:35.251514 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:35.251521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:35.251762 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:35.252666 (+ 904us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:35.252902 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.254534 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.256944 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.258355 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.260134 (+ 1779us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:35.260378 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:35.260492 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 03:09:35.260674 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":1869,"threads_started":1}
I20260928 03:09:35.261364 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:35.249440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:37469 (local address 127.0.0.1:36524)
0928 03:09:35.249962 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:35.249980 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:35.250090 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:35.253818 (+ 3728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:35.253822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:35.253850 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:35.254325 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:35.254334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.256331 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.256337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:35.258523 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:35.258538 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.260570 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.260577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:35.260745 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:35.260788 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 03:09:35.260879 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 03:09:35.261009 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":268,"threads_started":1}
I20260928 03:09:35.262247 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:35.249887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36524 (local address 127.6.192.1:37469)
0928 03:09:35.253298 (+ 3411us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:35.253304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:35.253327 (+ 23us) server_negotiation.cc:407] Connection header received
0928 03:09:35.253401 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:35.253406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:35.253629 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:35.253835 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:35.254494 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.256197 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.258768 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.260341 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.260485 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:35.260720 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:35.261741 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0928 03:09:35.261916 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3241,"thread_start_us":3119,"threads_started":1}
I20260928 03:09:35.263423 26703 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1911903aa14c728a3f308ecb3c4795 (DEFAULT_TABLE table=test-table [id=4e616c4b440e4624a133a58b86fe5506]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:09:35.265071 26703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1911903aa14c728a3f308ecb3c4795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:35.265142 26434 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1911903aa14c728a3f308ecb3c4795 (DEFAULT_TABLE table=test-table [id=4e616c4b440e4624a133a58b86fe5506]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:09:35.265024 26569 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1911903aa14c728a3f308ecb3c4795 (DEFAULT_TABLE table=test-table [id=4e616c4b440e4624a133a58b86fe5506]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:09:35.266474 26569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1911903aa14c728a3f308ecb3c4795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:35.266501 26434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1911903aa14c728a3f308ecb3c4795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:35.278108 26787 tablet_bootstrap.cc:492] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe: Bootstrap starting.
I20260928 03:09:35.280737 26788 tablet_bootstrap.cc:492] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3: Bootstrap starting.
I20260928 03:09:35.283519 26787 tablet_bootstrap.cc:654] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:35.284607 26788 tablet_bootstrap.cc:654] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:35.284896 26787 log.cc:824] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:35.285596 26789 tablet_bootstrap.cc:492] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd: Bootstrap starting.
I20260928 03:09:35.286007 26788 log.cc:824] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:35.289034 26787 tablet_bootstrap.cc:492] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe: No bootstrap required, opened a new log
I20260928 03:09:35.289343 26787 ts_tablet_manager.cc:1397] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260928 03:09:35.289443 26788 tablet_bootstrap.cc:492] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3: No bootstrap required, opened a new log
I20260928 03:09:35.289709 26788 ts_tablet_manager.cc:1397] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 03:09:35.289736 26789 tablet_bootstrap.cc:654] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:35.291038 26789 log.cc:824] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:35.294564 26789 tablet_bootstrap.cc:492] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd: No bootstrap required, opened a new log
I20260928 03:09:35.294872 26789 ts_tablet_manager.cc:1397] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260928 03:09:35.298146 26787 raft_consensus.cc:348] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c81d5556a44e69b5c52c75c3bf6ec3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46509 } } peers { permanent_uuid: "eab3d5e6d3cf4b0f95391cbaae71fefe" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37469 } } peers { permanent_uuid: "9965bd6bccd14e4399a5ce8bd18584dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35449 } }
I20260928 03:09:35.298552 26787 raft_consensus.cc:374] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:35.298648 26787 raft_consensus.cc:729] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eab3d5e6d3cf4b0f95391cbaae71fefe, State: Initialized, Role: FOLLOWER
I20260928 03:09:35.299327 26787 consensus_queue.cc:260] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c81d5556a44e69b5c52c75c3bf6ec3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46509 } } peers { permanent_uuid: "eab3d5e6d3cf4b0f95391cbaae71fefe" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37469 } } peers { permanent_uuid: "9965bd6bccd14e4399a5ce8bd18584dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35449 } }
I20260928 03:09:35.300398 26788 raft_consensus.cc:348] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c81d5556a44e69b5c52c75c3bf6ec3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46509 } } peers { permanent_uuid: "eab3d5e6d3cf4b0f95391cbaae71fefe" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37469 } } peers { permanent_uuid: "9965bd6bccd14e4399a5ce8bd18584dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35449 } }
I20260928 03:09:35.300859 26788 raft_consensus.cc:374] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:35.300949 26788 raft_consensus.cc:729] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72c81d5556a44e69b5c52c75c3bf6ec3, State: Initialized, Role: FOLLOWER
I20260928 03:09:35.301273 26787 ts_tablet_manager.cc:1428] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260928 03:09:35.301549 26788 consensus_queue.cc:260] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c81d5556a44e69b5c52c75c3bf6ec3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46509 } } peers { permanent_uuid: "eab3d5e6d3cf4b0f95391cbaae71fefe" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37469 } } peers { permanent_uuid: "9965bd6bccd14e4399a5ce8bd18584dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35449 } }
I20260928 03:09:35.303334 26768 heartbeater.cc:503] Master 127.6.192.62:46095 was elected leader, sending a full tablet report...
I20260928 03:09:35.303982 26788 ts_tablet_manager.cc:1428] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260928 03:09:35.306290 26789 raft_consensus.cc:348] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c81d5556a44e69b5c52c75c3bf6ec3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46509 } } peers { permanent_uuid: "eab3d5e6d3cf4b0f95391cbaae71fefe" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37469 } } peers { permanent_uuid: "9965bd6bccd14e4399a5ce8bd18584dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35449 } }
I20260928 03:09:35.306820 26789 raft_consensus.cc:374] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:35.306939 26789 raft_consensus.cc:729] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9965bd6bccd14e4399a5ce8bd18584dd, State: Initialized, Role: FOLLOWER
I20260928 03:09:35.307641 26789 consensus_queue.cc:260] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c81d5556a44e69b5c52c75c3bf6ec3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46509 } } peers { permanent_uuid: "eab3d5e6d3cf4b0f95391cbaae71fefe" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37469 } } peers { permanent_uuid: "9965bd6bccd14e4399a5ce8bd18584dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35449 } }
I20260928 03:09:35.308912 26634 heartbeater.cc:503] Master 127.6.192.62:46095 was elected leader, sending a full tablet report...
I20260928 03:09:35.309480 26789 ts_tablet_manager.cc:1428] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260928 03:09:35.310810 26795 raft_consensus.cc:482] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:09:35.311075 26795 raft_consensus.cc:504] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c81d5556a44e69b5c52c75c3bf6ec3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46509 } } peers { permanent_uuid: "eab3d5e6d3cf4b0f95391cbaae71fefe" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37469 } } peers { permanent_uuid: "9965bd6bccd14e4399a5ce8bd18584dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35449 } }
I20260928 03:09:35.312883 26795 leader_election.cc:288] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72c81d5556a44e69b5c52c75c3bf6ec3 (127.6.192.3:46509), eab3d5e6d3cf4b0f95391cbaae71fefe (127.6.192.1:37469)
W20260928 03:09:35.319242 26501 tablet.cc:2369] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:09:35.320696 26782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:35.313586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:52251 (local address 127.6.192.3:46509)
0928 03:09:35.313777 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:35.313784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:35.314033 (+ 249us) server_negotiation.cc:407] Connection header received
0928 03:09:35.314359 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:35.314365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:35.314457 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:35.314525 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:35.315158 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.316012 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.318117 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.319369 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.319553 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:35.319676 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:35.320272 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 03:09:35.320419 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 03:09:35.320731 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:35.314283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:54223 (local address 127.6.192.1:37469)
0928 03:09:35.314500 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:35.314506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:35.314693 (+ 187us) server_negotiation.cc:407] Connection header received
0928 03:09:35.314908 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:35.314913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:35.314983 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:35.315078 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:35.315608 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.316318 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.318232 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.319563 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.319708 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:35.319828 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:35.320334 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 03:09:35.320478 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 03:09:35.320842 26796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:35.313107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:46509 (local address 127.6.192.2:52251)
0928 03:09:35.313931 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:35.313955 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:35.314149 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:35.314692 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:35.314695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:35.314715 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:35.315017 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:35.315025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.316145 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.316153 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:35.317898 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:35.317913 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.319855 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.319863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:35.320098 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:35.320125 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 03:09:35.320243 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 03:09:35.320412 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":46,"thread_start_us":463,"threads_started":1}
I20260928 03:09:35.321205 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:35.313999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:37469 (local address 127.6.192.2:54223)
0928 03:09:35.314599 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:35.314612 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:35.314750 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:35.315090 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:35.315094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:35.315117 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:35.315426 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:35.315481 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.316460 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.316467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:35.318053 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:35.318071 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.319828 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.319839 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:35.320124 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:35.320171 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 03:09:35.320308 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 03:09:35.320478 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":55,"thread_start_us":308,"threads_started":1}
I20260928 03:09:35.321851 26723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1911903aa14c728a3f308ecb3c4795" candidate_uuid: "9965bd6bccd14e4399a5ce8bd18584dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72c81d5556a44e69b5c52c75c3bf6ec3" is_pre_election: true
I20260928 03:09:35.322207 26454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1911903aa14c728a3f308ecb3c4795" candidate_uuid: "9965bd6bccd14e4399a5ce8bd18584dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eab3d5e6d3cf4b0f95391cbaae71fefe" is_pre_election: true
I20260928 03:09:35.322484 26723 raft_consensus.cc:2454] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9965bd6bccd14e4399a5ce8bd18584dd in term 0.
I20260928 03:09:35.322777 26454 raft_consensus.cc:2454] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9965bd6bccd14e4399a5ce8bd18584dd in term 0.
I20260928 03:09:35.323311 26525 leader_election.cc:302] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72c81d5556a44e69b5c52c75c3bf6ec3, 9965bd6bccd14e4399a5ce8bd18584dd; no voters:
I20260928 03:09:35.323755 26795 raft_consensus.cc:2781] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:09:35.323930 26795 raft_consensus.cc:482] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:09:35.324060 26795 raft_consensus.cc:3037] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:35.326912 26795 raft_consensus.cc:504] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c81d5556a44e69b5c52c75c3bf6ec3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46509 } } peers { permanent_uuid: "eab3d5e6d3cf4b0f95391cbaae71fefe" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37469 } } peers { permanent_uuid: "9965bd6bccd14e4399a5ce8bd18584dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35449 } }
I20260928 03:09:35.327623 26795 leader_election.cc:288] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [CANDIDATE]: Term 1 election: Requested vote from peers 72c81d5556a44e69b5c52c75c3bf6ec3 (127.6.192.3:46509), eab3d5e6d3cf4b0f95391cbaae71fefe (127.6.192.1:37469)
I20260928 03:09:35.328140 26723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1911903aa14c728a3f308ecb3c4795" candidate_uuid: "9965bd6bccd14e4399a5ce8bd18584dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72c81d5556a44e69b5c52c75c3bf6ec3"
I20260928 03:09:35.328223 26454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1911903aa14c728a3f308ecb3c4795" candidate_uuid: "9965bd6bccd14e4399a5ce8bd18584dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eab3d5e6d3cf4b0f95391cbaae71fefe"
I20260928 03:09:35.328428 26454 raft_consensus.cc:3037] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:35.328388 26723 raft_consensus.cc:3037] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:35.331844 26723 raft_consensus.cc:2454] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9965bd6bccd14e4399a5ce8bd18584dd in term 1.
I20260928 03:09:35.331844 26454 raft_consensus.cc:2454] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9965bd6bccd14e4399a5ce8bd18584dd in term 1.
I20260928 03:09:35.332330 26525 leader_election.cc:302] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9965bd6bccd14e4399a5ce8bd18584dd, eab3d5e6d3cf4b0f95391cbaae71fefe; no voters:
I20260928 03:09:35.332681 26795 raft_consensus.cc:2781] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:35.333186 26795 raft_consensus.cc:686] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [term 1 LEADER]: Becoming Leader. State: Replica: 9965bd6bccd14e4399a5ce8bd18584dd, State: Running, Role: LEADER
I20260928 03:09:35.333756 26795 consensus_queue.cc:237] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c81d5556a44e69b5c52c75c3bf6ec3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46509 } } peers { permanent_uuid: "eab3d5e6d3cf4b0f95391cbaae71fefe" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37469 } } peers { permanent_uuid: "9965bd6bccd14e4399a5ce8bd18584dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35449 } }
I20260928 03:09:35.339135 26309 catalog_manager.cc:5495] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 9965bd6bccd14e4399a5ce8bd18584dd (127.6.192.2). New cstate: current_term: 1 leader_uuid: "9965bd6bccd14e4399a5ce8bd18584dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c81d5556a44e69b5c52c75c3bf6ec3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eab3d5e6d3cf4b0f95391cbaae71fefe" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9965bd6bccd14e4399a5ce8bd18584dd" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 35449 } health_report { overall_health: HEALTHY } } }
W20260928 03:09:35.361135 26635 tablet.cc:2369] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:09:35.368602 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:35.362750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.6.192.2:35449)
0928 03:09:35.362966 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:35.362972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:35.363051 (+ 79us) server_negotiation.cc:407] Connection header received
0928 03:09:35.363319 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:35.363327 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:35.363483 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:35.363663 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:09:35.364091 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.364809 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.366245 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.366675 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.366814 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:35.367368 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:09:35.367973 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:09:35.368152 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:35.368205 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 03:09:35.368292 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 03:09:35.377583 26454 raft_consensus.cc:1261] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe [term 1 FOLLOWER]: Refusing update from remote peer 9965bd6bccd14e4399a5ce8bd18584dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:09:35.377804 26723 raft_consensus.cc:1261] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3 [term 1 FOLLOWER]: Refusing update from remote peer 9965bd6bccd14e4399a5ce8bd18584dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:09:35.378901 26798 consensus_queue.cc:1035] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "eab3d5e6d3cf4b0f95391cbaae71fefe" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 37469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:09:35.379354 26795 consensus_queue.cc:1035] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "72c81d5556a44e69b5c52c75c3bf6ec3" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:09:35.398577 26803 mvcc.cc:205] Tried to move back new op lower bound from 7334154139135385600 to 7334154138972143616. Current Snapshot: MvccSnapshot[applied={T|T < 7334154139135385600 or (T in {7334154139135385600})}]
I20260928 03:09:35.411823 26309 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:09:35.412146 26309 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:09:35.415868 26309 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 997befca13d84a9390068d7c5dfcda0f: Sending DeleteTablet for 3 replicas of tablet 2b1911903aa14c728a3f308ecb3c4795
I20260928 03:09:35.417279 26703 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1911903aa14c728a3f308ecb3c4795 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:09:35 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:58460
I20260928 03:09:35.417573 26434 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1911903aa14c728a3f308ecb3c4795 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:09:35 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:36524
I20260928 03:09:35.419298 26569 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1911903aa14c728a3f308ecb3c4795 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:09:35 UTC) from {username='kudu', principal='kudu/127.6.192.62@KRBTEST.COM'} at 127.0.0.1:35338
I20260928 03:09:35.422667 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 26371
I20260928 03:09:35.434739 26810 tablet_replica.cc:330] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3: stopping tablet replica
I20260928 03:09:35.435361 26810 raft_consensus.cc:2229] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:09:35.436023 26810 raft_consensus.cc:2258] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:09:35.438592 26810 ts_tablet_manager.cc:1905] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:09:35.444063 26812 tablet_replica.cc:330] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd: stopping tablet replica
I20260928 03:09:35.444796 26811 tablet_replica.cc:330] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe: stopping tablet replica
I20260928 03:09:35.445434 26812 raft_consensus.cc:2229] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:09:35.446059 26811 raft_consensus.cc:2229] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:09:35.446897 26812 raft_consensus.cc:2258] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:09:35.447209 26810 ts_tablet_manager.cc:1918] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:09:35.447393 26810 log.cc:1196] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/2b1911903aa14c728a3f308ecb3c4795
I20260928 03:09:35.447754 26811 raft_consensus.cc:2258] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:09:35.447996 26810 ts_tablet_manager.cc:1939] T 2b1911903aa14c728a3f308ecb3c4795 P 72c81d5556a44e69b5c52c75c3bf6ec3: Deleting consensus metadata
I20260928 03:09:35.450467 26297 catalog_manager.cc:4841] TS 72c81d5556a44e69b5c52c75c3bf6ec3 (127.6.192.3:46509): tablet 2b1911903aa14c728a3f308ecb3c4795 (table test-table [id=4e616c4b440e4624a133a58b86fe5506]) successfully deleted
I20260928 03:09:35.451961 26811 ts_tablet_manager.cc:1905] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:09:35.451961 26812 ts_tablet_manager.cc:1905] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:09:35.459676 26811 ts_tablet_manager.cc:1918] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:09:35.459679 26812 ts_tablet_manager.cc:1918] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:09:35.459883 26812 log.cc:1196] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/2b1911903aa14c728a3f308ecb3c4795
I20260928 03:09:35.459877 26811 log.cc:1196] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/2b1911903aa14c728a3f308ecb3c4795
I20260928 03:09:35.460479 26811 ts_tablet_manager.cc:1939] T 2b1911903aa14c728a3f308ecb3c4795 P eab3d5e6d3cf4b0f95391cbaae71fefe: Deleting consensus metadata
I20260928 03:09:35.460491 26812 ts_tablet_manager.cc:1939] T 2b1911903aa14c728a3f308ecb3c4795 P 9965bd6bccd14e4399a5ce8bd18584dd: Deleting consensus metadata
I20260928 03:09:35.462323 26297 catalog_manager.cc:4841] TS eab3d5e6d3cf4b0f95391cbaae71fefe (127.6.192.1:37469): tablet 2b1911903aa14c728a3f308ecb3c4795 (table test-table [id=4e616c4b440e4624a133a58b86fe5506]) successfully deleted
I20260928 03:09:35.462647 26297 catalog_manager.cc:4841] TS 9965bd6bccd14e4399a5ce8bd18584dd (127.6.192.2:35449): tablet 2b1911903aa14c728a3f308ecb3c4795 (table test-table [id=4e616c4b440e4624a133a58b86fe5506]) successfully deleted
Sep 28 03:09:35 dist-test-slave-fjww krb5kdc[26265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.1@KRBTEST.COM
I20260928 03:09:35.480849 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:35.423881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36538 (local address 127.6.192.1:37469)
0928 03:09:35.424080 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:35.424088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:35.451506 (+ 27418us) server_negotiation.cc:407] Connection header received
0928 03:09:35.452525 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:35.452534 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:35.452645 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:35.452761 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:35.461063 (+ 8302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.462282 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.464476 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.464886 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.465020 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:35.474238 (+ 9218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:35.474264 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:35.474287 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:35.474349 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:35.477505 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:35.478341 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:35.478350 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:35.478352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:35.478502 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:35.478923 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:35.478934 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:35.478936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:35.479345 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:35.479696 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:35.480222 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 03:09:35.480476 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 03:09:35.481611 26495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:35.575129 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 26371
I20260928 03:09:35.593313 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 26506
Sep 28 03:09:35 dist-test-slave-fjww krb5kdc[26265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.2@KRBTEST.COM
I20260928 03:09:35.614331 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:35.594155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.6.192.2:35449)
0928 03:09:35.594337 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:35.594341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:35.594379 (+ 38us) server_negotiation.cc:407] Connection header received
0928 03:09:35.594605 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:35.594609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:35.594673 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:35.594903 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:35.596640 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.597472 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.599546 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.600017 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.600146 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:35.606249 (+ 6103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:35.606346 (+ 97us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:35.606370 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:35.606449 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:35.610513 (+ 4064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:35.611391 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:35.611399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:35.611402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:35.611766 (+ 364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:35.612301 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:35.612308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:35.612312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:35.612845 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:35.613335 (+ 490us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:35.613510 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 03:09:35.613774 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 03:09:35.616057 26629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:35.751053 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 26506
I20260928 03:09:35.769666 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 26640
Sep 28 03:09:35 dist-test-slave-fjww krb5kdc[26265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.3@KRBTEST.COM
I20260928 03:09:35.785830 26782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:35.770473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58466 (local address 127.6.192.3:46509)
0928 03:09:35.770639 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:35.770643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:35.770660 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:09:35.770790 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:35.770794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:35.770870 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:35.770969 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:35.772445 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.773245 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.774547 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:35.774943 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:35.775038 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:35.780069 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:35.780092 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:35.780108 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:35.780148 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:35.783044 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:35.784001 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:35.784009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:35.784011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:35.784155 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:35.784489 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:35.784492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:35.784494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:35.784840 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:35.785109 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:35.785399 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 03:09:35.785570 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 03:09:35.786566 26763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:35.886034 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 26640
I20260928 03:09:35.902913 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 26278
I20260928 03:09:35.904279 26340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:36.019097 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 26278
2026-09-28T03:09:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4484 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T03:09:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:09:36Z Disabled control of system clock
I20260928 03:09:36.101804 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:36949
--webserver_interface=127.6.192.62
--webserver_port=0
--builtin_ntp_servers=127.6.192.20:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:36949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 03:09:36.456879 26821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:36.457224 26821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:36.457327 26821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:36.465204 26821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:09:36.465353 26821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:36.465396 26821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:36.465466 26821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:09:36.465514 26821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:09:36.475397 26821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:36949
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:36949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:36.476766 26821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:36.478669 26821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:36.492203 26826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:36.492239 26827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:36.492271 26829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:36.493280 26821 server_base.cc:1034] running on GCE node
I20260928 03:09:36.494164 26821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:36.495900 26821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:36.497134 26821 hybrid_clock.cc:648] HybridClock initialized: now 1790564976497062 us; error 68 us; skew 500 ppm
I20260928 03:09:36.497606 26821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:09:36.500250 26821 webserver.cc:460] Webserver started at http://127.6.192.62:34035/ using document root <none> and password file <none>
I20260928 03:09:36.501020 26821 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:36.501163 26821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:36.501516 26821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:36.504696 26821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "52d55903d9a6464a9801f8f5c90f544f"
format_stamp: "Formatted at 2026-09-28 03:09:36 on dist-test-slave-fjww"
server_key: "bb46593d5287480e203c0ef8620eea0b"
server_key_iv: "ee3ece891a4db66a7e7894afa11f0b5f"
server_key_version: "encryptionkey@0"
I20260928 03:09:36.505599 26821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "52d55903d9a6464a9801f8f5c90f544f"
format_stamp: "Formatted at 2026-09-28 03:09:36 on dist-test-slave-fjww"
server_key: "bb46593d5287480e203c0ef8620eea0b"
server_key_iv: "ee3ece891a4db66a7e7894afa11f0b5f"
server_key_version: "encryptionkey@0"
I20260928 03:09:36.511190 26821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 03:09:36.515574 26835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:36.517131 26821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:09:36.517342 26821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "52d55903d9a6464a9801f8f5c90f544f"
format_stamp: "Formatted at 2026-09-28 03:09:36 on dist-test-slave-fjww"
server_key: "bb46593d5287480e203c0ef8620eea0b"
server_key_iv: "ee3ece891a4db66a7e7894afa11f0b5f"
server_key_version: "encryptionkey@0"
I20260928 03:09:36.517589 26821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:36.561612 26821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:36.562659 26821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:36.563047 26821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:36.584375 26821 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:36949
I20260928 03:09:36.584409 26886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:36949 every 8 connection(s)
I20260928 03:09:36.585959 26821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:09:36.590018 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 26821
I20260928 03:09:36.590197 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:09:36.590371 26887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:36.590723 6912 external_mini_cluster.cc:1428] Setting key 916c731778ad62240a1624d24824c021
I20260928 03:09:36.607321 26887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f: Bootstrap starting.
I20260928 03:09:36.607879 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:36.592890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33804 (local address 127.6.192.62:36949)
0928 03:09:36.594170 (+ 1280us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:36.594191 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:36.594253 (+ 62us) server_negotiation.cc:407] Connection header received
0928 03:09:36.596220 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:36.596255 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:36.596980 (+ 725us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:36.597485 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:36.598158 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:36.600260 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:36.601724 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:36.602882 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:36.603092 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:36.604604 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:36.604616 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:36.604639 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:36.605462 (+ 823us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:36.605590 (+ 128us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:36.605621 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:36.605756 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:36.606029 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 03:09:36.606417 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":418,"threads_started":1}
I20260928 03:09:36.611073 26887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:36.612260 26887 log.cc:824] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:36.615638 26887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f: No bootstrap required, opened a new log
I20260928 03:09:36.624086 26887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d55903d9a6464a9801f8f5c90f544f" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36949 } }
I20260928 03:09:36.624513 26887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:36.624593 26887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52d55903d9a6464a9801f8f5c90f544f, State: Initialized, Role: FOLLOWER
I20260928 03:09:36.625142 26887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d55903d9a6464a9801f8f5c90f544f" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36949 } }
I20260928 03:09:36.625377 26887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:09:36.625494 26887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:09:36.625628 26887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:36.628350 26887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d55903d9a6464a9801f8f5c90f544f" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36949 } }
I20260928 03:09:36.628862 26887 leader_election.cc:302] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52d55903d9a6464a9801f8f5c90f544f; no voters:
I20260928 03:09:36.629402 26887 leader_election.cc:288] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:09:36.629565 26892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:36.630314 26892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f [term 1 LEADER]: Becoming Leader. State: Replica: 52d55903d9a6464a9801f8f5c90f544f, State: Running, Role: LEADER
I20260928 03:09:36.631006 26892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d55903d9a6464a9801f8f5c90f544f" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36949 } }
I20260928 03:09:36.631747 26887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:09:36.633817 26894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52d55903d9a6464a9801f8f5c90f544f. Latest consensus state: current_term: 1 leader_uuid: "52d55903d9a6464a9801f8f5c90f544f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d55903d9a6464a9801f8f5c90f544f" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36949 } } }
I20260928 03:09:36.634506 26894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:36.634379 26893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52d55903d9a6464a9801f8f5c90f544f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d55903d9a6464a9801f8f5c90f544f" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36949 } } }
I20260928 03:09:36.634761 26893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:36.636790 26900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:09:36.642323 26900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:09:36.652026 26900 catalog_manager.cc:1347] Generated new cluster ID: d506b19646f94b63949111eaaa98a6f1
I20260928 03:09:36.652179 26900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:09:36.671732 26900 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:09:36.673167 26900 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:09:36.680864 26900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f: Generated new TSK 0
I20260928 03:09:36.682026 26900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:09:36.696453 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.1:0
--local_ip_for_outbound_sockets=127.6.192.1
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:36949
--builtin_ntp_servers=127.6.192.20:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 03:09:37.060935 26911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:37.061488 26911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:37.061705 26911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:37.070263 26911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:37.070433 26911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:37.070600 26911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.1
I20260928 03:09:37.080130 26911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.192.1
--webserver_port=0
--tserver_master_addrs=127.6.192.62:36949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.1
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:37.081544 26911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:37.083544 26911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:37.098253 26917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:37.098701 26919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:37.098779 26916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:37.099082 26911 server_base.cc:1034] running on GCE node
I20260928 03:09:37.099944 26911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:37.100950 26911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:37.102231 26911 hybrid_clock.cc:648] HybridClock initialized: now 1790564977102144 us; error 81 us; skew 500 ppm
I20260928 03:09:37.102653 26911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:09:37.105424 26911 webserver.cc:460] Webserver started at http://127.6.192.1:41303/ using document root <none> and password file <none>
I20260928 03:09:37.106209 26911 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:37.106362 26911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:37.106827 26911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:37.110146 26911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-0/data/instance:
uuid: "f12b497a1169428e8f822ffb50c84130"
format_stamp: "Formatted at 2026-09-28 03:09:37 on dist-test-slave-fjww"
server_key: "6f5a7206d3d590fd2ec1c11a3b92c7fd"
server_key_iv: "e50aaff9a33c6b69b121371a62a72671"
server_key_version: "encryptionkey@0"
I20260928 03:09:37.111020 26911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance:
uuid: "f12b497a1169428e8f822ffb50c84130"
format_stamp: "Formatted at 2026-09-28 03:09:37 on dist-test-slave-fjww"
server_key: "6f5a7206d3d590fd2ec1c11a3b92c7fd"
server_key_iv: "e50aaff9a33c6b69b121371a62a72671"
server_key_version: "encryptionkey@0"
I20260928 03:09:37.116873 26911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 03:09:37.120963 26925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:37.122663 26911 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:09:37.122816 26911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-0/wal
uuid: "f12b497a1169428e8f822ffb50c84130"
format_stamp: "Formatted at 2026-09-28 03:09:37 on dist-test-slave-fjww"
server_key: "6f5a7206d3d590fd2ec1c11a3b92c7fd"
server_key_iv: "e50aaff9a33c6b69b121371a62a72671"
server_key_version: "encryptionkey@0"
I20260928 03:09:37.123109 26911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:37.134948 26911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:37.135885 26911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:37.136204 26911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:37.137362 26911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:37.138962 26911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:37.139088 26911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:37.139238 26911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:37.139338 26911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:37.174190 26911 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.1:33317
I20260928 03:09:37.174245 27037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.1:33317 every 8 connection(s)
I20260928 03:09:37.175607 26911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-0/data/info.pb
I20260928 03:09:37.185712 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 26911
I20260928 03:09:37.186020 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-0/wal/instance
I20260928 03:09:37.186619 6912 external_mini_cluster.cc:1428] Setting key 4570582cf9ffbad704ebeb3011b8edd7
I20260928 03:09:37.189873 27040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:37.178304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.1:47037 (local address 127.6.192.62:36949)
0928 03:09:37.178743 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:37.178748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:37.180206 (+ 1458us) server_negotiation.cc:407] Connection header received
0928 03:09:37.182088 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:37.182093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:37.182166 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:37.182273 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:37.183548 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:37.184353 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:37.186094 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:37.186811 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:37.186964 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:37.188709 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:37.188715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:37.188718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:37.188737 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:37.188918 (+ 181us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:37.188942 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:37.189045 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:37.189380 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 03:09:37.189489 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":216,"threads_started":1}
I20260928 03:09:37.190351 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.2:0
--local_ip_for_outbound_sockets=127.6.192.2
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:36949
--builtin_ntp_servers=127.6.192.20:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 03:09:37.191035 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:37.178901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36949 (local address 127.6.192.1:47037)
0928 03:09:37.180039 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:37.180084 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:37.181786 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:37.182558 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:37.182568 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:37.182679 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:37.183313 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:37.183331 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:37.184519 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:37.184523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:37.185853 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:37.185867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:37.187082 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:37.187087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:37.187383 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:37.188281 (+ 898us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:09:37.188284 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:37.188348 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:09:37.188473 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:37.189071 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:37.189084 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:37.189093 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 03:09:37.189296 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 03:09:37.189667 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":312,"threads_started":1}
I20260928 03:09:37.192811 27038 heartbeater.cc:347] Connected to a master server at 127.6.192.62:36949
I20260928 03:09:37.193177 27038 heartbeater.cc:464] Registering TS with master...
I20260928 03:09:37.194137 27038 heartbeater.cc:511] Master 127.6.192.62:36949 requested a full tablet report, sending...
I20260928 03:09:37.196539 26852 ts_manager.cc:194] Registered new tserver with Master: f12b497a1169428e8f822ffb50c84130 (127.6.192.1:33317)
I20260928 03:09:37.198047 26852 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.192.1:47037
W20260928 03:09:37.536837 27042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:37.537230 27042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:37.537356 27042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:37.544739 27042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:37.544857 27042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:37.544973 27042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.2
I20260928 03:09:37.554260 27042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.192.2
--webserver_port=0
--tserver_master_addrs=127.6.192.62:36949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.2
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:37.555621 27042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:37.557372 27042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:37.571333 27047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:37.571403 27050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:37.571508 27048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:37.572827 27042 server_base.cc:1034] running on GCE node
I20260928 03:09:37.573602 27042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:37.574646 27042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:37.575927 27042 hybrid_clock.cc:648] HybridClock initialized: now 1790564977575819 us; error 61 us; skew 500 ppm
I20260928 03:09:37.576277 27042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:09:37.578920 27042 webserver.cc:460] Webserver started at http://127.6.192.2:35363/ using document root <none> and password file <none>
I20260928 03:09:37.579772 27042 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:37.579921 27042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:37.580261 27042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:37.583310 27042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-1/data/instance:
uuid: "3e3635896ea6423a9cfccce0b608fc49"
format_stamp: "Formatted at 2026-09-28 03:09:37 on dist-test-slave-fjww"
server_key: "06c6e213835380f7bdba1e7497d89497"
server_key_iv: "29b0e28a5c52e25a8f5e59388d12ceca"
server_key_version: "encryptionkey@0"
I20260928 03:09:37.584273 27042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance:
uuid: "3e3635896ea6423a9cfccce0b608fc49"
format_stamp: "Formatted at 2026-09-28 03:09:37 on dist-test-slave-fjww"
server_key: "06c6e213835380f7bdba1e7497d89497"
server_key_iv: "29b0e28a5c52e25a8f5e59388d12ceca"
server_key_version: "encryptionkey@0"
I20260928 03:09:37.589754 27042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:09:37.593694 27056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:37.595285 27042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 03:09:37.595552 27042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-1/wal
uuid: "3e3635896ea6423a9cfccce0b608fc49"
format_stamp: "Formatted at 2026-09-28 03:09:37 on dist-test-slave-fjww"
server_key: "06c6e213835380f7bdba1e7497d89497"
server_key_iv: "29b0e28a5c52e25a8f5e59388d12ceca"
server_key_version: "encryptionkey@0"
I20260928 03:09:37.595790 27042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:37.621147 27042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:37.622139 27042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:37.622454 27042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:37.623627 27042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:37.625226 27042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:37.625353 27042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:37.625487 27042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:37.625578 27042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:37.662227 27042 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.2:33857
I20260928 03:09:37.662323 27168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.2:33857 every 8 connection(s)
I20260928 03:09:37.663650 27042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-1/data/info.pb
I20260928 03:09:37.669525 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 27042
I20260928 03:09:37.669705 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-1/wal/instance
I20260928 03:09:37.670250 6912 external_mini_cluster.cc:1428] Setting key 2cecc839a979aadd9790345ebdf2bebd
I20260928 03:09:37.673813 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.192.3:0
--local_ip_for_outbound_sockets=127.6.192.3
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:36949
--builtin_ntp_servers=127.6.192.20:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 03:09:37.679710 27040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:37.666475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.2:42035 (local address 127.6.192.62:36949)
0928 03:09:37.666711 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:37.666718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:37.668258 (+ 1540us) server_negotiation.cc:407] Connection header received
0928 03:09:37.670666 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:37.670672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:37.670754 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:37.670872 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:37.672261 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:37.673151 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:37.676085 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:37.676534 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:37.676662 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:37.678721 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:37.678729 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:37.678744 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:37.678768 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:37.678805 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:37.678819 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:37.678900 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:37.679241 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 03:09:37.679350 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 03:09:37.681320 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:37.666985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36949 (local address 127.6.192.2:42035)
0928 03:09:37.668074 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:37.668114 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:37.670283 (+ 2169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:37.671123 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:37.671133 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:37.671301 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:37.672063 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:37.672082 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:37.673323 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:37.673330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:37.675726 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:37.675747 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:37.676820 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:37.676830 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:37.677220 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:37.678225 (+ 1005us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:09:37.678231 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:37.678291 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:09:37.678427 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:37.678931 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:37.678946 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:37.678957 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 03:09:37.679160 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 03:09:37.679715 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":401,"threads_started":1}
I20260928 03:09:37.683121 27169 heartbeater.cc:347] Connected to a master server at 127.6.192.62:36949
I20260928 03:09:37.683580 27169 heartbeater.cc:464] Registering TS with master...
I20260928 03:09:37.684433 27169 heartbeater.cc:511] Master 127.6.192.62:36949 requested a full tablet report, sending...
I20260928 03:09:37.686175 26852 ts_manager.cc:194] Registered new tserver with Master: 3e3635896ea6423a9cfccce0b608fc49 (127.6.192.2:33857)
I20260928 03:09:37.687554 26852 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.192.2:42035
W20260928 03:09:38.030426 27172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:38.030746 27172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:38.030843 27172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:38.039520 27172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:38.039680 27172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:38.039882 27172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.192.3
I20260928 03:09:38.049561 27172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.192.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.192.3
--webserver_port=0
--tserver_master_addrs=127.6.192.62:36949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.192.3
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:38.050984 27172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:38.053011 27172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:38.067365 27177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:38.067431 27178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:38.067482 27180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:38.068799 27172 server_base.cc:1034] running on GCE node
I20260928 03:09:38.069509 27172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:38.070556 27172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:38.071791 27172 hybrid_clock.cc:648] HybridClock initialized: now 1790564978071738 us; error 48 us; skew 500 ppm
I20260928 03:09:38.072154 27172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 03:09:38.074687 27172 webserver.cc:460] Webserver started at http://127.6.192.3:41593/ using document root <none> and password file <none>
I20260928 03:09:38.075428 27172 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:38.075627 27172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:38.075976 27172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:38.078940 27172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-2/data/instance:
uuid: "070d774e59d1463b920dcd6bf1dcc786"
format_stamp: "Formatted at 2026-09-28 03:09:38 on dist-test-slave-fjww"
server_key: "87010f9a8cf437f024537a93a555b1c5"
server_key_iv: "7b9b6bb6e1a2f7954ba8b6575669f101"
server_key_version: "encryptionkey@0"
I20260928 03:09:38.079849 27172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance:
uuid: "070d774e59d1463b920dcd6bf1dcc786"
format_stamp: "Formatted at 2026-09-28 03:09:38 on dist-test-slave-fjww"
server_key: "87010f9a8cf437f024537a93a555b1c5"
server_key_iv: "7b9b6bb6e1a2f7954ba8b6575669f101"
server_key_version: "encryptionkey@0"
I20260928 03:09:38.085299 27172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 03:09:38.089247 27186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:38.090876 27172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 03:09:38.091116 27172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-2/wal
uuid: "070d774e59d1463b920dcd6bf1dcc786"
format_stamp: "Formatted at 2026-09-28 03:09:38 on dist-test-slave-fjww"
server_key: "87010f9a8cf437f024537a93a555b1c5"
server_key_iv: "7b9b6bb6e1a2f7954ba8b6575669f101"
server_key_version: "encryptionkey@0"
I20260928 03:09:38.091403 27172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:38.105374 27172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:38.106369 27172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:38.106691 27172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:38.107861 27172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 03:09:38.109733 27172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 03:09:38.109869 27172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:38.110020 27172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 03:09:38.110116 27172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:38.148341 27172 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.3:33379
I20260928 03:09:38.148391 27298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.3:33379 every 8 connection(s)
I20260928 03:09:38.149716 27172 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-2/data/info.pb
I20260928 03:09:38.155495 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 27172
I20260928 03:09:38.155675 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-2/wal/instance
I20260928 03:09:38.156255 6912 external_mini_cluster.cc:1428] Setting key ad2b25b0a6de1dda0e7950b98f7f9bef
I20260928 03:09:38.164134 27040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:38.152620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:42569 (local address 127.6.192.62:36949)
0928 03:09:38.152836 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:38.152841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:38.154395 (+ 1554us) server_negotiation.cc:407] Connection header received
0928 03:09:38.156981 (+ 2586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:38.156988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:38.157086 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:38.157198 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:38.158492 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.159308 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.161201 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.161624 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.161733 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:38.163311 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:38.163316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:38.163320 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:38.163341 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:38.163375 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:38.163388 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:38.163529 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:38.163763 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 03:09:38.163864 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 03:09:38.165280 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:38.153103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:36949 (local address 127.6.192.3:42569)
0928 03:09:38.154202 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:38.154251 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:38.156617 (+ 2366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:38.157471 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:38.157480 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:38.157585 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:09:38.158325 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:38.158342 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.159515 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.159520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:38.161033 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:38.161048 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.161827 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.161832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:38.162121 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:38.162946 (+ 825us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 03:09:38.162949 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:09:38.163001 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 03:09:38.163092 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:09:38.163544 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:09:38.163553 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:09:38.163560 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 03:09:38.163724 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 03:09:38.164123 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":321,"threads_started":1}
I20260928 03:09:38.166916 27299 heartbeater.cc:347] Connected to a master server at 127.6.192.62:36949
I20260928 03:09:38.167264 27299 heartbeater.cc:464] Registering TS with master...
I20260928 03:09:38.168229 27299 heartbeater.cc:511] Master 127.6.192.62:36949 requested a full tablet report, sending...
I20260928 03:09:38.169919 26852 ts_manager.cc:194] Registered new tserver with Master: 070d774e59d1463b920dcd6bf1dcc786 (127.6.192.3:33379)
I20260928 03:09:38.171078 26852 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.192.3:42569
I20260928 03:09:38.172106 6912 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 03:09:38.184074 27040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:38.177294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.6.192.62:36949)
0928 03:09:38.177506 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:38.177512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:38.177966 (+ 454us) server_negotiation.cc:407] Connection header received
0928 03:09:38.178115 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:38.178122 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:38.178202 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:38.178342 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:38.178833 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.179904 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.181449 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.182002 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.182171 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:38.183335 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:38.183339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:38.183343 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:38.183363 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:38.183397 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:38.183408 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:38.183569 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:38.183749 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 03:09:38.183833 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 03:09:38.188352 26851 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 03:09:38.196342 26851 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 03:09:38.202040 27038 heartbeater.cc:503] Master 127.6.192.62:36949 was elected leader, sending a full tablet report...
I20260928 03:09:38.223331 27315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:38.213855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57794 (local address 127.6.192.2:33857)
0928 03:09:38.214925 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:38.214934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:38.215753 (+ 819us) server_negotiation.cc:407] Connection header received
0928 03:09:38.215897 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:38.215905 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:38.216178 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:38.216387 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:38.217016 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.218341 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.220596 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.222171 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.222303 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:38.222551 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:38.222856 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 03:09:38.223069 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":893,"thread_start_us":771,"threads_started":1}
I20260928 03:09:38.223372 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:38.210170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:33317 (local address 127.0.0.1:46428)
0928 03:09:38.212517 (+ 2347us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:38.212569 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:38.212736 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:38.214345 (+ 1609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:38.214356 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:38.214390 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:38.214842 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:38.214856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.216740 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.216746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:38.219764 (+ 3018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:38.219785 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.221604 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.221610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:38.221903 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:38.222013 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 03:09:38.222723 (+ 710us) client_negotiation.cc:240] Negotiation successful
0928 03:09:38.222957 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2059,"mutex_wait_us":87,"thread_start_us":2836,"threads_started":1}
I20260928 03:09:38.223610 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:38.210824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34536 (local address 127.6.192.3:33379)
0928 03:09:38.211882 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:38.211896 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:38.213482 (+ 1586us) server_negotiation.cc:407] Connection header received
0928 03:09:38.213772 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:38.213780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:38.214003 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:38.214146 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:38.215609 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.217356 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.220297 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.221888 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.222074 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:38.222373 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:38.222966 (+ 593us) server_negotiation.cc:299] Negotiation successful
0928 03:09:38.223149 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":700,"threads_started":1}
I20260928 03:09:38.224174 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:38.214977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:33857 (local address 127.0.0.1:57794)
0928 03:09:38.215610 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:38.215629 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:38.215751 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:38.216387 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:38.216392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:38.216418 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:38.216826 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:38.216838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.218527 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.218532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:38.220347 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:38.220362 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.222468 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.222474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:38.222639 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:38.222675 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 03:09:38.223680 (+ 1005us) client_negotiation.cc:240] Negotiation successful
0928 03:09:38.223828 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":327,"threads_started":1}
I20260928 03:09:38.224323 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:38.210483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46428 (local address 127.6.192.1:33317)
0928 03:09:38.213374 (+ 2891us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:38.213385 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:38.213433 (+ 48us) server_negotiation.cc:407] Connection header received
0928 03:09:38.213549 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:38.213557 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:38.213846 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:38.214053 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:38.215093 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.216577 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.220007 (+ 3430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.221374 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.221571 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:38.222323 (+ 752us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:38.223715 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0928 03:09:38.223932 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2710,"thread_start_us":2569,"threads_started":1}
I20260928 03:09:38.224577 27311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:38.210316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.3:33379 (local address 127.0.0.1:34536)
0928 03:09:38.213370 (+ 3054us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:38.213388 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:38.213585 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:38.214431 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:38.214437 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:38.214464 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:38.215396 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:38.215411 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.217567 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.217573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:38.220099 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:38.220111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.222528 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.222535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:38.222738 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:38.222770 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 03:09:38.223963 (+ 1193us) client_negotiation.cc:240] Negotiation successful
0928 03:09:38.224119 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2872,"mutex_wait_us":5,"spinlock_wait_cycles":2432,"thread_start_us":1808,"threads_started":1}
I20260928 03:09:38.228919 27234 tablet_service.cc:1467] Processing CreateTablet for tablet b203d2d5e8e24d96a623a6fe32bcf7f0 (DEFAULT_TABLE table=test-table [id=f807de1abe0b4f22808989df7b0c0e95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:09:38.228840 27104 tablet_service.cc:1467] Processing CreateTablet for tablet b203d2d5e8e24d96a623a6fe32bcf7f0 (DEFAULT_TABLE table=test-table [id=f807de1abe0b4f22808989df7b0c0e95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:09:38.229445 26973 tablet_service.cc:1467] Processing CreateTablet for tablet b203d2d5e8e24d96a623a6fe32bcf7f0 (DEFAULT_TABLE table=test-table [id=f807de1abe0b4f22808989df7b0c0e95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 03:09:38.230301 27234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b203d2d5e8e24d96a623a6fe32bcf7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:38.230441 27104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b203d2d5e8e24d96a623a6fe32bcf7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:38.230855 26973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b203d2d5e8e24d96a623a6fe32bcf7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:38.243274 27317 tablet_bootstrap.cc:492] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786: Bootstrap starting.
I20260928 03:09:38.246846 27317 tablet_bootstrap.cc:654] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:38.247004 27318 tablet_bootstrap.cc:492] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49: Bootstrap starting.
I20260928 03:09:38.248298 27319 tablet_bootstrap.cc:492] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130: Bootstrap starting.
I20260928 03:09:38.248375 27317 log.cc:824] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:38.251397 27318 tablet_bootstrap.cc:654] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:38.252202 27319 tablet_bootstrap.cc:654] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:38.252825 27317 tablet_bootstrap.cc:492] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786: No bootstrap required, opened a new log
I20260928 03:09:38.252808 27318 log.cc:824] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:38.253111 27317 ts_tablet_manager.cc:1397] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260928 03:09:38.253466 27319 log.cc:824] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:38.256104 27318 tablet_bootstrap.cc:492] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49: No bootstrap required, opened a new log
I20260928 03:09:38.256379 27318 ts_tablet_manager.cc:1397] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 03:09:38.257599 27319 tablet_bootstrap.cc:492] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130: No bootstrap required, opened a new log
I20260928 03:09:38.257881 27319 ts_tablet_manager.cc:1397] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 03:09:38.264540 27317 raft_consensus.cc:348] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12b497a1169428e8f822ffb50c84130" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33317 } } peers { permanent_uuid: "070d774e59d1463b920dcd6bf1dcc786" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33379 } } peers { permanent_uuid: "3e3635896ea6423a9cfccce0b608fc49" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33857 } }
I20260928 03:09:38.265089 27317 raft_consensus.cc:374] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:38.265233 27317 raft_consensus.cc:729] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070d774e59d1463b920dcd6bf1dcc786, State: Initialized, Role: FOLLOWER
I20260928 03:09:38.265992 27317 consensus_queue.cc:260] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12b497a1169428e8f822ffb50c84130" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33317 } } peers { permanent_uuid: "070d774e59d1463b920dcd6bf1dcc786" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33379 } } peers { permanent_uuid: "3e3635896ea6423a9cfccce0b608fc49" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33857 } }
I20260928 03:09:38.267306 27299 heartbeater.cc:503] Master 127.6.192.62:36949 was elected leader, sending a full tablet report...
I20260928 03:09:38.267184 27319 raft_consensus.cc:348] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12b497a1169428e8f822ffb50c84130" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33317 } } peers { permanent_uuid: "070d774e59d1463b920dcd6bf1dcc786" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33379 } } peers { permanent_uuid: "3e3635896ea6423a9cfccce0b608fc49" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33857 } }
I20260928 03:09:38.267349 27318 raft_consensus.cc:348] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12b497a1169428e8f822ffb50c84130" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33317 } } peers { permanent_uuid: "070d774e59d1463b920dcd6bf1dcc786" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33379 } } peers { permanent_uuid: "3e3635896ea6423a9cfccce0b608fc49" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33857 } }
I20260928 03:09:38.267751 27319 raft_consensus.cc:374] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:38.267859 27319 raft_consensus.cc:729] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12b497a1169428e8f822ffb50c84130, State: Initialized, Role: FOLLOWER
I20260928 03:09:38.267943 27318 raft_consensus.cc:374] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:38.268069 27318 raft_consensus.cc:729] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e3635896ea6423a9cfccce0b608fc49, State: Initialized, Role: FOLLOWER
I20260928 03:09:38.268150 27317 ts_tablet_manager.cc:1428] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786: Time spent starting tablet: real 0.015s user 0.013s sys 0.002s
I20260928 03:09:38.268667 27319 consensus_queue.cc:260] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12b497a1169428e8f822ffb50c84130" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33317 } } peers { permanent_uuid: "070d774e59d1463b920dcd6bf1dcc786" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33379 } } peers { permanent_uuid: "3e3635896ea6423a9cfccce0b608fc49" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33857 } }
I20260928 03:09:38.268793 27318 consensus_queue.cc:260] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12b497a1169428e8f822ffb50c84130" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33317 } } peers { permanent_uuid: "070d774e59d1463b920dcd6bf1dcc786" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33379 } } peers { permanent_uuid: "3e3635896ea6423a9cfccce0b608fc49" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33857 } }
I20260928 03:09:38.270653 27319 ts_tablet_manager.cc:1428] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260928 03:09:38.271692 27169 heartbeater.cc:503] Master 127.6.192.62:36949 was elected leader, sending a full tablet report...
I20260928 03:09:38.272442 27318 ts_tablet_manager.cc:1428] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260928 03:09:38.401535 27300 tablet.cc:2369] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:09:38.415915 27170 tablet.cc:2369] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 03:09:38.428174 27039 tablet.cc:2369] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:09:38.496753 27323 raft_consensus.cc:482] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 03:09:38.497105 27323 raft_consensus.cc:504] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12b497a1169428e8f822ffb50c84130" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33317 } } peers { permanent_uuid: "070d774e59d1463b920dcd6bf1dcc786" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33379 } } peers { permanent_uuid: "3e3635896ea6423a9cfccce0b608fc49" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33857 } }
I20260928 03:09:38.498893 27323 leader_election.cc:288] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12b497a1169428e8f822ffb50c84130 (127.6.192.1:33317), 3e3635896ea6423a9cfccce0b608fc49 (127.6.192.2:33857)
I20260928 03:09:38.505857 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:38.499700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:60613 (local address 127.6.192.1:33317)
0928 03:09:38.499866 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:38.499870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:38.499890 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:09:38.500028 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:38.500033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:38.500130 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:38.500259 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:38.500701 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.501641 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.504099 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.504925 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.505071 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:38.505170 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:38.505472 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 03:09:38.505595 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 03:09:38.505864 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:38.499457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.1:33317 (local address 127.6.192.3:60613)
0928 03:09:38.499695 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:38.499718 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:38.499869 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:38.500304 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:38.500309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:38.500338 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:38.500536 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:38.500545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.501945 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.501954 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:38.503937 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:38.503949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.505081 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.505087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:38.505292 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:38.505316 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 03:09:38.505420 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 03:09:38.505561 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 03:09:38.506593 27315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:38.500265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.192.3:54565 (local address 127.6.192.2:33857)
0928 03:09:38.500525 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:38.500534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:38.501188 (+ 654us) server_negotiation.cc:407] Connection header received
0928 03:09:38.501332 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:38.501338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:38.501421 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:38.501514 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 03:09:38.502082 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.503120 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.504890 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.505651 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.505803 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:38.505904 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:38.506194 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 03:09:38.506316 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 03:09:38.507251 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:38.500082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.2:33857 (local address 127.6.192.3:54565)
0928 03:09:38.501070 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0928 03:09:38.501090 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 03:09:38.501204 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:09:38.501548 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:09:38.501551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:09:38.501571 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 03:09:38.501901 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:38.501910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.503286 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.503290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:38.504674 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:09:38.504685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.505828 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.505835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:09:38.506019 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:38.506048 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 03:09:38.506837 (+ 789us) client_negotiation.cc:240] Negotiation successful
0928 03:09:38.506969 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"thread_start_us":671,"threads_started":1}
I20260928 03:09:38.507157 26993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b203d2d5e8e24d96a623a6fe32bcf7f0" candidate_uuid: "070d774e59d1463b920dcd6bf1dcc786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12b497a1169428e8f822ffb50c84130" is_pre_election: true
I20260928 03:09:38.507817 26993 raft_consensus.cc:2454] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 070d774e59d1463b920dcd6bf1dcc786 in term 0.
I20260928 03:09:38.508472 27187 leader_election.cc:302] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 070d774e59d1463b920dcd6bf1dcc786, f12b497a1169428e8f822ffb50c84130; no voters:
I20260928 03:09:38.508435 27124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b203d2d5e8e24d96a623a6fe32bcf7f0" candidate_uuid: "070d774e59d1463b920dcd6bf1dcc786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e3635896ea6423a9cfccce0b608fc49" is_pre_election: true
I20260928 03:09:38.508898 27323 raft_consensus.cc:2781] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 03:09:38.508937 27124 raft_consensus.cc:2454] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 070d774e59d1463b920dcd6bf1dcc786 in term 0.
I20260928 03:09:38.509044 27323 raft_consensus.cc:482] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 03:09:38.509193 27323 raft_consensus.cc:3037] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:38.511541 27323 raft_consensus.cc:504] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12b497a1169428e8f822ffb50c84130" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33317 } } peers { permanent_uuid: "070d774e59d1463b920dcd6bf1dcc786" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33379 } } peers { permanent_uuid: "3e3635896ea6423a9cfccce0b608fc49" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33857 } }
I20260928 03:09:38.512188 27323 leader_election.cc:288] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [CANDIDATE]: Term 1 election: Requested vote from peers f12b497a1169428e8f822ffb50c84130 (127.6.192.1:33317), 3e3635896ea6423a9cfccce0b608fc49 (127.6.192.2:33857)
I20260928 03:09:38.512692 26993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b203d2d5e8e24d96a623a6fe32bcf7f0" candidate_uuid: "070d774e59d1463b920dcd6bf1dcc786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12b497a1169428e8f822ffb50c84130"
I20260928 03:09:38.512837 27124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b203d2d5e8e24d96a623a6fe32bcf7f0" candidate_uuid: "070d774e59d1463b920dcd6bf1dcc786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e3635896ea6423a9cfccce0b608fc49"
I20260928 03:09:38.512898 26993 raft_consensus.cc:3037] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:38.513087 27124 raft_consensus.cc:3037] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:38.515136 26993 raft_consensus.cc:2454] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 070d774e59d1463b920dcd6bf1dcc786 in term 1.
I20260928 03:09:38.515324 27124 raft_consensus.cc:2454] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 070d774e59d1463b920dcd6bf1dcc786 in term 1.
I20260928 03:09:38.515635 27187 leader_election.cc:302] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 070d774e59d1463b920dcd6bf1dcc786, f12b497a1169428e8f822ffb50c84130; no voters:
I20260928 03:09:38.515959 27323 raft_consensus.cc:2781] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:38.516415 27323 raft_consensus.cc:686] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [term 1 LEADER]: Becoming Leader. State: Replica: 070d774e59d1463b920dcd6bf1dcc786, State: Running, Role: LEADER
I20260928 03:09:38.516891 27323 consensus_queue.cc:237] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12b497a1169428e8f822ffb50c84130" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33317 } } peers { permanent_uuid: "070d774e59d1463b920dcd6bf1dcc786" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33379 } } peers { permanent_uuid: "3e3635896ea6423a9cfccce0b608fc49" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33857 } }
I20260928 03:09:38.521807 26852 catalog_manager.cc:5495] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 reported cstate change: term changed from 0 to 1, leader changed from <none> to 070d774e59d1463b920dcd6bf1dcc786 (127.6.192.3). New cstate: current_term: 1 leader_uuid: "070d774e59d1463b920dcd6bf1dcc786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12b497a1169428e8f822ffb50c84130" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "070d774e59d1463b920dcd6bf1dcc786" member_type: VOTER last_known_addr { host: "127.6.192.3" port: 33379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e3635896ea6423a9cfccce0b608fc49" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33857 } health_report { overall_health: UNKNOWN } } }
I20260928 03:09:38.592262 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:38.586076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.6.192.3:33379)
0928 03:09:38.586274 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:38.586278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:38.586295 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:09:38.586385 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:38.586389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:38.586468 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:38.586603 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:09:38.587036 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.587841 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.589938 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.590361 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.590472 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:38.591051 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:09:38.591676 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:09:38.591789 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:38.591924 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 03:09:38.592011 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 03:09:38.601224 27124 raft_consensus.cc:1261] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49 [term 1 FOLLOWER]: Refusing update from remote peer 070d774e59d1463b920dcd6bf1dcc786: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:09:38.601222 26993 raft_consensus.cc:1261] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130 [term 1 FOLLOWER]: Refusing update from remote peer 070d774e59d1463b920dcd6bf1dcc786: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:09:38.602612 27323 consensus_queue.cc:1035] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e3635896ea6423a9cfccce0b608fc49" member_type: VOTER last_known_addr { host: "127.6.192.2" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:09:38.602967 27327 consensus_queue.cc:1035] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12b497a1169428e8f822ffb50c84130" member_type: VOTER last_known_addr { host: "127.6.192.1" port: 33317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:09:38.615656 27330 mvcc.cc:205] Tried to move back new op lower bound from 7334154152338329600 to 7334154152008871936. Current Snapshot: MvccSnapshot[applied={T|T < 7334154152338329600}]
I20260928 03:09:38.631183 26852 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:09:38.631562 26852 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 03:09:38.635805 26852 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52d55903d9a6464a9801f8f5c90f544f: Sending DeleteTablet for 3 replicas of tablet b203d2d5e8e24d96a623a6fe32bcf7f0
I20260928 03:09:38.637298 27234 tablet_service.cc:1514] Processing DeleteTablet for tablet b203d2d5e8e24d96a623a6fe32bcf7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:09:38 UTC) from {username='slave'} at 127.0.0.1:34536
I20260928 03:09:38.637295 27104 tablet_service.cc:1514] Processing DeleteTablet for tablet b203d2d5e8e24d96a623a6fe32bcf7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:09:38 UTC) from {username='slave'} at 127.0.0.1:57794
I20260928 03:09:38.638376 26973 tablet_service.cc:1514] Processing DeleteTablet for tablet b203d2d5e8e24d96a623a6fe32bcf7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 03:09:38 UTC) from {username='slave'} at 127.0.0.1:46428
I20260928 03:09:38.638520 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 26911
I20260928 03:09:38.654098 27341 tablet_replica.cc:330] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786: stopping tablet replica
I20260928 03:09:38.654781 27341 raft_consensus.cc:2229] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [term 1 LEADER]: Raft consensus shutting down.
I20260928 03:09:38.655817 27341 raft_consensus.cc:2258] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:09:38.656996 27342 tablet_replica.cc:330] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130: stopping tablet replica
I20260928 03:09:38.657621 27342 raft_consensus.cc:2229] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:09:38.658226 27342 raft_consensus.cc:2258] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:09:38.660717 27342 ts_tablet_manager.cc:1905] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:09:38.665721 27341 ts_tablet_manager.cc:1905] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:09:38.671675 27342 ts_tablet_manager.cc:1918] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:09:38.671890 27342 log.cc:1196] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-0/wal/wals/b203d2d5e8e24d96a623a6fe32bcf7f0
I20260928 03:09:38.672417 27342 ts_tablet_manager.cc:1939] T b203d2d5e8e24d96a623a6fe32bcf7f0 P f12b497a1169428e8f822ffb50c84130: Deleting consensus metadata
I20260928 03:09:38.674377 26836 catalog_manager.cc:4841] TS f12b497a1169428e8f822ffb50c84130 (127.6.192.1:33317): tablet b203d2d5e8e24d96a623a6fe32bcf7f0 (table test-table [id=f807de1abe0b4f22808989df7b0c0e95]) successfully deleted
I20260928 03:09:38.677191 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:38.640586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46438 (local address 127.6.192.1:33317)
0928 03:09:38.640816 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:38.640826 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:38.651624 (+ 10798us) server_negotiation.cc:407] Connection header received
0928 03:09:38.652252 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:38.652262 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:38.652411 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:38.652501 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:38.656374 (+ 3873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.658066 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.660838 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.661743 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.661889 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:38.675355 (+ 13466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:38.675368 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:38.675387 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:38.676212 (+ 825us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:38.676283 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:38.676311 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:38.676426 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:38.676702 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 03:09:38.676844 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 03:09:38.678467 27033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:38.678761 27340 tablet_replica.cc:330] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49: stopping tablet replica
I20260928 03:09:38.679329 27340 raft_consensus.cc:2229] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 03:09:38.679930 27340 raft_consensus.cc:2258] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 03:09:38.683821 27341 ts_tablet_manager.cc:1918] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:09:38.684033 27341 log.cc:1196] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-2/wal/wals/b203d2d5e8e24d96a623a6fe32bcf7f0
I20260928 03:09:38.684993 27341 ts_tablet_manager.cc:1939] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 070d774e59d1463b920dcd6bf1dcc786: Deleting consensus metadata
I20260928 03:09:38.686726 27340 ts_tablet_manager.cc:1905] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 03:09:38.687057 26838 catalog_manager.cc:4841] TS 070d774e59d1463b920dcd6bf1dcc786 (127.6.192.3:33379): tablet b203d2d5e8e24d96a623a6fe32bcf7f0 (table test-table [id=f807de1abe0b4f22808989df7b0c0e95]) successfully deleted
I20260928 03:09:38.694947 27340 ts_tablet_manager.cc:1918] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 03:09:38.695148 27340 log.cc:1196] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49: Deleting WAL directory at /tmp/dist-test-task97hosG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790564644636890-6912-0/minicluster-data/ts-1/wal/wals/b203d2d5e8e24d96a623a6fe32bcf7f0
I20260928 03:09:38.695752 27340 ts_tablet_manager.cc:1939] T b203d2d5e8e24d96a623a6fe32bcf7f0 P 3e3635896ea6423a9cfccce0b608fc49: Deleting consensus metadata
I20260928 03:09:38.697789 26836 catalog_manager.cc:4841] TS 3e3635896ea6423a9cfccce0b608fc49 (127.6.192.2:33857): tablet b203d2d5e8e24d96a623a6fe32bcf7f0 (table test-table [id=f807de1abe0b4f22808989df7b0c0e95]) successfully deleted
I20260928 03:09:38.783636 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 26911
I20260928 03:09:38.808015 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 27042
I20260928 03:09:38.817310 27315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:38.809109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.6.192.2:33857)
0928 03:09:38.809331 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:38.809339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:38.809389 (+ 50us) server_negotiation.cc:407] Connection header received
0928 03:09:38.809588 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:38.809600 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:38.809730 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:38.809865 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:38.810293 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.811496 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.813025 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.813773 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.813930 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:38.815134 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:38.815148 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:38.815170 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:38.816291 (+ 1121us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:38.816362 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:38.816405 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:38.816539 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:38.816714 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 03:09:38.816848 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 03:09:38.818174 27164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:38.908690 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 27042
I20260928 03:09:38.926934 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 27172
I20260928 03:09:38.934859 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:38.927783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34556 (local address 127.6.192.3:33379)
0928 03:09:38.927960 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:38.927966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:38.928083 (+ 117us) server_negotiation.cc:407] Connection header received
0928 03:09:38.928339 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:38.928345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:38.928447 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:38.928576 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:38.929019 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.929821 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.931153 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:38.931952 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:38.932115 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:38.933370 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:38.933379 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:38.933395 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:38.934017 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:38.934070 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:38.934092 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:38.934239 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:38.934376 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 03:09:38.934494 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 03:09:38.935674 27294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:39.056222 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 27172
I20260928 03:09:39.075850 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 26821
I20260928 03:09:39.076941 26882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:39.198585 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 26821
2026-09-28T03:09:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3184 ms)
[----------] 6 tests from EncryptionPolicyTest (23308 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:09:39 dist-test-slave-fjww krb5kdc[27350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:09:39 dist-test-slave-fjww krb5kdc[27350](info): set up 2 sockets
Sep 28 03:09:39 dist-test-slave-fjww krb5kdc[27350](info): commencing operation
krb5kdc: starting...
W20260928 03:09:40.372123 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:09:40 dist-test-slave-fjww krb5kdc[27350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:09:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:09:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:09:40.652040 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43415
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:41.017966 27363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:41.018271 27363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:41.018442 27363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:41.026027 27363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:09:41.026160 27363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:41.026259 27363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:41.026368 27363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:09:41.026435 27363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:09:41.036867 27363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43415
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:41.038292 27363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:41.039819 27363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:41.053570 27368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:41.053594 27371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:41.053624 27369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:41.053833 27363 server_base.cc:1034] running on GCE node
I20260928 03:09:41.055145 27363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:41.056821 27363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:41.058079 27363 hybrid_clock.cc:648] HybridClock initialized: now 1790564981058009 us; error 71 us; skew 500 ppm
I20260928 03:09:41.058463 27363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:41 dist-test-slave-fjww krb5kdc[27350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:41.065201 27363 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:09:41.066978 27363 webserver.cc:460] Webserver started at http://127.0.0.1:46287/ using document root <none> and password file <none>
I20260928 03:09:41.067829 27363 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:41.067988 27363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:41.068310 27363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:41.071406 27363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "689fb4d7bedc47489149b86f15ec1175"
format_stamp: "Formatted at 2026-09-28 03:09:41 on dist-test-slave-fjww"
server_key: "4878ad04f49b4b47adc522a0431a1c28"
server_key_iv: "60a5c226612dcee77eebdfeefafa7bc4"
server_key_version: "encryptionkey@0"
I20260928 03:09:41.072311 27363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "689fb4d7bedc47489149b86f15ec1175"
format_stamp: "Formatted at 2026-09-28 03:09:41 on dist-test-slave-fjww"
server_key: "4878ad04f49b4b47adc522a0431a1c28"
server_key_iv: "60a5c226612dcee77eebdfeefafa7bc4"
server_key_version: "encryptionkey@0"
I20260928 03:09:41.077984 27363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:09:41.082056 27378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:41.083806 27363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:09:41.084025 27363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "689fb4d7bedc47489149b86f15ec1175"
format_stamp: "Formatted at 2026-09-28 03:09:41 on dist-test-slave-fjww"
server_key: "4878ad04f49b4b47adc522a0431a1c28"
server_key_iv: "60a5c226612dcee77eebdfeefafa7bc4"
server_key_version: "encryptionkey@0"
I20260928 03:09:41.084244 27363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:41.113197 27363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:41.118547 27363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:41.118906 27363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:41.140040 27363 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43415
I20260928 03:09:41.140084 27429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43415 every 8 connection(s)
I20260928 03:09:41.141825 27363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:09:41.142221 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 27363
I20260928 03:09:41.142453 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:09:41.143286 6912 external_mini_cluster.cc:1428] Setting key 6252872edeb1616d87ef088a69303602
I20260928 03:09:41.147223 27430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 03:09:41 dist-test-slave-fjww krb5kdc[27350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 03:09:41.162461 27430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175: Bootstrap starting.
I20260928 03:09:41.166642 27430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:41.168220 27430 log.cc:824] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:41.171546 27433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:41.145538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54161 (local address 127.0.0.1:43415)
0928 03:09:41.146483 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:41.146500 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:41.146554 (+ 54us) server_negotiation.cc:407] Connection header received
0928 03:09:41.149230 (+ 2676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:41.149263 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:41.149845 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:41.150353 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:41.151907 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:41.154274 (+ 2367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:41.155829 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:41.157085 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:41.157296 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:41.162337 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:41.162347 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:41.162365 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:41.162410 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:41.165897 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:41.167187 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:41.167198 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:41.167205 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:41.167399 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:41.168032 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:41.168036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:41.168038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:41.168684 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:41.169089 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:41.169675 (+ 586us) server_negotiation.cc:299] Negotiation successful
0928 03:09:41.170115 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":399,"threads_started":1}
I20260928 03:09:41.172027 27430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175: No bootstrap required, opened a new log
I20260928 03:09:41.175992 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:09:41.180348 27430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "689fb4d7bedc47489149b86f15ec1175" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43415 } }
I20260928 03:09:41.180799 27430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:41.180954 27430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 689fb4d7bedc47489149b86f15ec1175, State: Initialized, Role: FOLLOWER
I20260928 03:09:41.181663 27430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "689fb4d7bedc47489149b86f15ec1175" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43415 } }
I20260928 03:09:41.181965 27430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:09:41.182145 27430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:09:41.182358 27430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:41.185328 27430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "689fb4d7bedc47489149b86f15ec1175" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43415 } }
I20260928 03:09:41.185910 27430 leader_election.cc:302] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 689fb4d7bedc47489149b86f15ec1175; no voters:
I20260928 03:09:41.186607 27430 leader_election.cc:288] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:09:41.186797 27440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:41.188112 27440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175 [term 1 LEADER]: Becoming Leader. State: Replica: 689fb4d7bedc47489149b86f15ec1175, State: Running, Role: LEADER
I20260928 03:09:41.188817 27440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "689fb4d7bedc47489149b86f15ec1175" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43415 } }
I20260928 03:09:41.189338 27430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:09:41.191665 27441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "689fb4d7bedc47489149b86f15ec1175" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "689fb4d7bedc47489149b86f15ec1175" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43415 } } }
I20260928 03:09:41.191956 27442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 689fb4d7bedc47489149b86f15ec1175. Latest consensus state: current_term: 1 leader_uuid: "689fb4d7bedc47489149b86f15ec1175" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "689fb4d7bedc47489149b86f15ec1175" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43415 } } }
I20260928 03:09:41.192221 27442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175 [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:41.192530 27441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175 [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:41.193486 27448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:09:41.202258 27433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:41.180900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38091 (local address 127.0.0.1:43415)
0928 03:09:41.181123 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:41.181129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:41.181685 (+ 556us) server_negotiation.cc:407] Connection header received
0928 03:09:41.181833 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:41.181839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:41.181932 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:41.182055 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:41.183774 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:41.184984 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:41.186336 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:41.187273 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:41.188190 (+ 917us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:41.191052 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:41.191062 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:41.191069 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:41.191127 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:41.196090 (+ 4963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:41.197617 (+ 1527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:41.197625 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:41.197632 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:41.197877 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:41.198648 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:41.198656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:41.198661 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:41.199225 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:41.201142 (+ 1917us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:41.201378 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 03:09:41.201640 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 03:09:41.211048 27448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:09:41.220871 27448 catalog_manager.cc:1347] Generated new cluster ID: 80bfb5e7067f442d9c9745cfc7f0c0a7
I20260928 03:09:41.221122 27448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:09:41.238884 27448 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:09:41.240844 27448 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:09:41.252300 27448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 689fb4d7bedc47489149b86f15ec1175: Generated new TSK 0
I20260928 03:09:41.253430 27448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:09:41.271189 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 27363
I20260928 03:09:41.273267 27425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:41.368232 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 27363
W20260928 03:09:41.387305 27348 connection.cc:535] client connection to 127.0.0.1:43415 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43415): Connection reset by peer (error 104)
2026-09-28T03:09:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2159 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:09:41 dist-test-slave-fjww krb5kdc[27462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:09:41 dist-test-slave-fjww krb5kdc[27462](info): set up 2 sockets
Sep 28 03:09:41 dist-test-slave-fjww krb5kdc[27462](info): commencing operation
krb5kdc: starting...
W20260928 03:09:42.552084 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:09:42 dist-test-slave-fjww krb5kdc[27462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:09:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:09:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:09:42.836985 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37319
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:43.187767 27475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:43.188143 27475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:43.188309 27475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:43.195940 27475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:09:43.196115 27475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:43.196226 27475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:43.196345 27475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:09:43.196432 27475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:09:43.206959 27475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37319
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:43.208415 27475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:43.209779 27475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:43.222756 27480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:43.222703 27483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:43.223124 27481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:43.224170 27475 server_base.cc:1034] running on GCE node
I20260928 03:09:43.225001 27475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:43.226595 27475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:43.227833 27475 hybrid_clock.cc:648] HybridClock initialized: now 1790564983227764 us; error 76 us; skew 500 ppm
I20260928 03:09:43.228212 27475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:43 dist-test-slave-fjww krb5kdc[27462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:43.231704 27475 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:09:43.233464 27475 webserver.cc:460] Webserver started at http://127.0.0.1:37963/ using document root <none> and password file <none>
I20260928 03:09:43.234241 27475 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:43.234397 27475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:43.234724 27475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:43.237979 27475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "35b29d46b82d4a2fb6d636d0009969d8"
format_stamp: "Formatted at 2026-09-28 03:09:43 on dist-test-slave-fjww"
server_key: "69ade842fd7a9271d9ddd4904114f9e0"
server_key_iv: "9a52abbaf125cc1f47a5ca6cf4046a50"
server_key_version: "encryptionkey@0"
I20260928 03:09:43.238794 27475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "35b29d46b82d4a2fb6d636d0009969d8"
format_stamp: "Formatted at 2026-09-28 03:09:43 on dist-test-slave-fjww"
server_key: "69ade842fd7a9271d9ddd4904114f9e0"
server_key_iv: "9a52abbaf125cc1f47a5ca6cf4046a50"
server_key_version: "encryptionkey@0"
I20260928 03:09:43.244684 27475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 03:09:43.248729 27490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:43.250517 27475 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 03:09:43.250723 27475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "35b29d46b82d4a2fb6d636d0009969d8"
format_stamp: "Formatted at 2026-09-28 03:09:43 on dist-test-slave-fjww"
server_key: "69ade842fd7a9271d9ddd4904114f9e0"
server_key_iv: "9a52abbaf125cc1f47a5ca6cf4046a50"
server_key_version: "encryptionkey@0"
I20260928 03:09:43.250954 27475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:43.272230 27475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:43.276738 27475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:43.277047 27475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:43.297864 27475 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37319
I20260928 03:09:43.297995 27541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37319 every 8 connection(s)
I20260928 03:09:43.299541 27475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:09:43.304029 27542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:43.305131 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 27475
I20260928 03:09:43.305351 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:09:43.305970 6912 external_mini_cluster.cc:1428] Setting key 4387c268d750b85bf3f7feba6b3ed3ca
I20260928 03:09:43.318481 27542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8: Bootstrap starting.
I20260928 03:09:43.320571 27545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:43.308133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54741 (local address 127.0.0.1:37319)
0928 03:09:43.309042 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:43.309056 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:43.309103 (+ 47us) server_negotiation.cc:407] Connection header received
0928 03:09:43.311398 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:43.311430 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:43.311986 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:43.312427 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:43.313051 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:43.314321 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:43.315796 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:43.316539 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:43.316742 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:43.318128 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:43.318136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:43.318152 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:43.318769 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:43.318851 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:43.318877 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:43.319058 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:43.319332 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 03:09:43.319562 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":371,"threads_started":1}
I20260928 03:09:43.322135 27542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:43.323575 27542 log.cc:824] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:43.325193 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:09:43.327160 27542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8: No bootstrap required, opened a new log
I20260928 03:09:43.337129 27545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:43.329781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45933 (local address 127.0.0.1:37319)
0928 03:09:43.329997 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:43.330004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:43.330430 (+ 426us) server_negotiation.cc:407] Connection header received
0928 03:09:43.330592 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:43.330598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:43.330704 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:43.330830 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:43.331411 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:43.332627 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:43.333971 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:43.334757 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:43.334924 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:43.336210 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:43.336226 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:43.336231 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:43.336262 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:43.336321 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:43.336351 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:43.336508 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:43.336661 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 03:09:43.336731 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 03:09:43.338413 27542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b29d46b82d4a2fb6d636d0009969d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37319 } }
I20260928 03:09:43.338936 27542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:43.339064 27542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b29d46b82d4a2fb6d636d0009969d8, State: Initialized, Role: FOLLOWER
I20260928 03:09:43.339833 27542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b29d46b82d4a2fb6d636d0009969d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37319 } }
I20260928 03:09:43.340124 27542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:09:43.340287 27542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:09:43.340483 27542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:43.342967 27542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b29d46b82d4a2fb6d636d0009969d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37319 } }
I20260928 03:09:43.343664 27542 leader_election.cc:302] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35b29d46b82d4a2fb6d636d0009969d8; no voters:
I20260928 03:09:43.344326 27542 leader_election.cc:288] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:09:43.344472 27552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:43.345211 27552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8 [term 1 LEADER]: Becoming Leader. State: Replica: 35b29d46b82d4a2fb6d636d0009969d8, State: Running, Role: LEADER
I20260928 03:09:43.345816 27552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b29d46b82d4a2fb6d636d0009969d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37319 } }
I20260928 03:09:43.346644 27542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:09:43.349046 27554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35b29d46b82d4a2fb6d636d0009969d8. Latest consensus state: current_term: 1 leader_uuid: "35b29d46b82d4a2fb6d636d0009969d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b29d46b82d4a2fb6d636d0009969d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37319 } } }
I20260928 03:09:43.349680 27554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8 [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:43.349094 27553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35b29d46b82d4a2fb6d636d0009969d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b29d46b82d4a2fb6d636d0009969d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37319 } } }
I20260928 03:09:43.351321 27553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8 [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:43.356081 27560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:09:43.361781 27560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:09:43.371399 27560 catalog_manager.cc:1347] Generated new cluster ID: 4cf88fb2173047e8ae188970e202fb0f
I20260928 03:09:43.372082 27560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:09:43.399114 27560 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:09:43.400553 27560 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:09:43.417636 27560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35b29d46b82d4a2fb6d636d0009969d8: Generated new TSK 0
I20260928 03:09:43.418834 27560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:09:43.439646 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 27475
W20260928 03:09:43.441323 27537 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54741
W20260928 03:09:43.442271 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 27475: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:09:43.442507 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 27475
W20260928 03:09:43.461916 27460 connection.cc:535] client connection to 127.0.0.1:37319 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37319): Connection reset by peer (error 104)
2026-09-28T03:09:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2074 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:09:43 dist-test-slave-fjww krb5kdc[27573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:09:43 dist-test-slave-fjww krb5kdc[27573](info): set up 2 sockets
Sep 28 03:09:43 dist-test-slave-fjww krb5kdc[27573](info): commencing operation
krb5kdc: starting...
W20260928 03:09:44.592108 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:09:44 dist-test-slave-fjww krb5kdc[27573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:09:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:09:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:09:44.876513 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35759
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:45.242197 27586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:45.242534 27586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:45.242713 27586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:45.250623 27586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:09:45.250753 27586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:45.250877 27586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:45.250979 27586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:09:45.251063 27586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:09:45.261911 27586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35759
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:45.263501 27586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:45.264986 27586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:45.278316 27591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:45.278360 27592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:45.278316 27594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:45.278995 27586 server_base.cc:1034] running on GCE node
I20260928 03:09:45.279963 27586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:45.281648 27586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:45.282931 27586 hybrid_clock.cc:648] HybridClock initialized: now 1790564985282880 us; error 32 us; skew 500 ppm
I20260928 03:09:45.283350 27586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:45 dist-test-slave-fjww krb5kdc[27573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:45.287285 27586 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:09:45.289027 27586 webserver.cc:460] Webserver started at http://127.0.0.1:37391/ using document root <none> and password file <none>
I20260928 03:09:45.289907 27586 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:45.290067 27586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:45.290470 27586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:45.293977 27586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "a20b0e55bb204e55b2043b5847da5429"
format_stamp: "Formatted at 2026-09-28 03:09:45 on dist-test-slave-fjww"
server_key: "12b09d398743d431accc786524283067"
server_key_iv: "45306b9188b5aa819c565266abf7d2a6"
server_key_version: "encryptionkey@0"
I20260928 03:09:45.294880 27586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "a20b0e55bb204e55b2043b5847da5429"
format_stamp: "Formatted at 2026-09-28 03:09:45 on dist-test-slave-fjww"
server_key: "12b09d398743d431accc786524283067"
server_key_iv: "45306b9188b5aa819c565266abf7d2a6"
server_key_version: "encryptionkey@0"
I20260928 03:09:45.301039 27586 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 03:09:45.305245 27601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:45.306958 27586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260928 03:09:45.307210 27586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "a20b0e55bb204e55b2043b5847da5429"
format_stamp: "Formatted at 2026-09-28 03:09:45 on dist-test-slave-fjww"
server_key: "12b09d398743d431accc786524283067"
server_key_iv: "45306b9188b5aa819c565266abf7d2a6"
server_key_version: "encryptionkey@0"
I20260928 03:09:45.307500 27586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:45.313925 27586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:45.318702 27586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:45.319079 27586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:45.339119 27586 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35759
I20260928 03:09:45.339166 27652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35759 every 8 connection(s)
I20260928 03:09:45.340771 27586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:09:45.342706 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 27586
I20260928 03:09:45.342914 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:09:45.343511 6912 external_mini_cluster.cc:1428] Setting key 389ab713ad69fe1b86e6524f0e021a4d
I20260928 03:09:45.345898 27653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:45.352744 27656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:45.345402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36129 (local address 127.0.0.1:35759)
0928 03:09:45.346409 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:45.346421 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:45.346465 (+ 44us) server_negotiation.cc:407] Connection header received
0928 03:09:45.348598 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:45.348625 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:45.349035 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:45.349428 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:45.350741 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:45.350748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:45.350764 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:45.350782 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:45.350985 (+ 203us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:45.351010 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:45.351204 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:45.351350 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 03:09:45.351655 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":573,"threads_started":1}
I20260928 03:09:45.357163 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:09:45.359381 27653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429: Bootstrap starting.
I20260928 03:09:45.363154 27653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:45.364645 27653 log.cc:824] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:45.364873 27656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:45.361830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51625 (local address 127.0.0.1:35759)
0928 03:09:45.362096 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:45.362106 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:45.362563 (+ 457us) server_negotiation.cc:407] Connection header received
0928 03:09:45.362795 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:45.362800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:45.362865 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:45.362962 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:45.364056 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:45.364061 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:45.364064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:45.364080 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:45.364120 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:45.364136 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:45.364239 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:45.364402 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 03:09:45.364454 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 03:09:45.368382 27653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429: No bootstrap required, opened a new log
I20260928 03:09:45.376999 27653 raft_consensus.cc:348] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20b0e55bb204e55b2043b5847da5429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35759 } }
I20260928 03:09:45.377444 27653 raft_consensus.cc:374] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:45.377610 27653 raft_consensus.cc:729] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20b0e55bb204e55b2043b5847da5429, State: Initialized, Role: FOLLOWER
I20260928 03:09:45.378270 27653 consensus_queue.cc:260] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20b0e55bb204e55b2043b5847da5429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35759 } }
I20260928 03:09:45.378508 27653 raft_consensus.cc:388] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:09:45.378666 27653 raft_consensus.cc:482] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:09:45.378854 27653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:45.381763 27653 raft_consensus.cc:504] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20b0e55bb204e55b2043b5847da5429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35759 } }
I20260928 03:09:45.382328 27653 leader_election.cc:302] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a20b0e55bb204e55b2043b5847da5429; no voters:
I20260928 03:09:45.382946 27653 leader_election.cc:288] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:09:45.383119 27663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:45.383885 27663 raft_consensus.cc:686] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429 [term 1 LEADER]: Becoming Leader. State: Replica: a20b0e55bb204e55b2043b5847da5429, State: Running, Role: LEADER
I20260928 03:09:45.384541 27663 consensus_queue.cc:237] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20b0e55bb204e55b2043b5847da5429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35759 } }
I20260928 03:09:45.385311 27653 sys_catalog.cc:564] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:09:45.388450 27664 sys_catalog.cc:455] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a20b0e55bb204e55b2043b5847da5429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20b0e55bb204e55b2043b5847da5429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35759 } } }
I20260928 03:09:45.388837 27665 sys_catalog.cc:455] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a20b0e55bb204e55b2043b5847da5429. Latest consensus state: current_term: 1 leader_uuid: "a20b0e55bb204e55b2043b5847da5429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20b0e55bb204e55b2043b5847da5429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35759 } } }
I20260928 03:09:45.389055 27665 sys_catalog.cc:458] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429 [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:45.389058 27664 sys_catalog.cc:458] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429 [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:45.394531 27670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:09:45.402083 27670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:09:45.412175 27670 catalog_manager.cc:1347] Generated new cluster ID: c0b73fc331ec4ed1ad61df55b7a9cda5
I20260928 03:09:45.412359 27670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:09:45.442519 27670 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:09:45.443069 27670 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:09:45.468400 27670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a20b0e55bb204e55b2043b5847da5429: Generated new TSK 0
I20260928 03:09:45.469548 27670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:09:45.483804 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 27586
W20260928 03:09:45.485062 27648 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36129
W20260928 03:09:45.485962 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 27586: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:09:45.486145 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 27586
2026-09-28T03:09:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2041 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:09:45 dist-test-slave-fjww krb5kdc[27684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:09:45 dist-test-slave-fjww krb5kdc[27684](info): set up 2 sockets
Sep 28 03:09:45 dist-test-slave-fjww krb5kdc[27684](info): commencing operation
krb5kdc: starting...
W20260928 03:09:46.692790 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:09:46 dist-test-slave-fjww krb5kdc[27684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:09:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:09:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:09:46.984289 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38397
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:47.337894 27697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:47.338207 27697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:47.338387 27697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:47.346155 27697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:09:47.346290 27697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:47.346381 27697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:47.346498 27697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:09:47.346561 27697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:09:47.357245 27697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:47.358719 27697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:47.360209 27697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:47.373433 27703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:47.373464 27705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:47.373376 27702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:47.374235 27697 server_base.cc:1034] running on GCE node
I20260928 03:09:47.375120 27697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:47.376924 27697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:47.378197 27697 hybrid_clock.cc:648] HybridClock initialized: now 1790564987378149 us; error 53 us; skew 500 ppm
I20260928 03:09:47.378612 27697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:47 dist-test-slave-fjww krb5kdc[27684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:47.385872 27697 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:09:47.388190 27697 webserver.cc:460] Webserver started at http://127.0.0.1:34553/ using document root <none> and password file <none>
I20260928 03:09:47.389020 27697 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:47.389187 27697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:47.389551 27697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:47.393075 27697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "2c01835430d94fa39248e487996b2833"
format_stamp: "Formatted at 2026-09-28 03:09:47 on dist-test-slave-fjww"
server_key: "754fa8e29fae478c548459d63f6dc5f2"
server_key_iv: "2af41a08ef6eb5cf8fc9c96216ada12e"
server_key_version: "encryptionkey@0"
I20260928 03:09:47.393982 27697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "2c01835430d94fa39248e487996b2833"
format_stamp: "Formatted at 2026-09-28 03:09:47 on dist-test-slave-fjww"
server_key: "754fa8e29fae478c548459d63f6dc5f2"
server_key_iv: "2af41a08ef6eb5cf8fc9c96216ada12e"
server_key_version: "encryptionkey@0"
I20260928 03:09:47.399997 27697 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 03:09:47.404286 27712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:47.405912 27697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:09:47.406184 27697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "2c01835430d94fa39248e487996b2833"
format_stamp: "Formatted at 2026-09-28 03:09:47 on dist-test-slave-fjww"
server_key: "754fa8e29fae478c548459d63f6dc5f2"
server_key_iv: "2af41a08ef6eb5cf8fc9c96216ada12e"
server_key_version: "encryptionkey@0"
I20260928 03:09:47.406432 27697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:47.421324 27697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:47.426324 27697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:47.426656 27697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:47.448473 27697 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38397
I20260928 03:09:47.448524 27763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38397 every 8 connection(s)
I20260928 03:09:47.450155 27697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:09:47.452653 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 27697
I20260928 03:09:47.452852 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:09:47.453426 6912 external_mini_cluster.cc:1428] Setting key 5f6582c8b5846da67eae73fc1547efd8
I20260928 03:09:47.455406 27764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 03:09:47 dist-test-slave-fjww krb5kdc[27684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 03:09:47.468420 27764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833: Bootstrap starting.
I20260928 03:09:47.471886 27764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:47.473088 27764 log.cc:824] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:47.476312 27764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833: No bootstrap required, opened a new log
I20260928 03:09:47.477412 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:47.455360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51001 (local address 127.0.0.1:38397)
0928 03:09:47.456309 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:47.456321 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:47.456364 (+ 43us) server_negotiation.cc:407] Connection header received
0928 03:09:47.458065 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:47.458089 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:47.458533 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:47.458897 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:47.460957 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:47.462448 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:47.463977 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:47.464700 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:47.464934 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:47.469577 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:47.469586 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:47.469602 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:47.469649 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:47.472978 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:47.473862 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:47.473869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:47.473876 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:47.474063 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:47.474490 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:47.474494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:47.474496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:47.475048 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:47.475364 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:47.475853 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 03:09:47.476328 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":433,"threads_started":1}
I20260928 03:09:47.481719 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:09:47.485221 27764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c01835430d94fa39248e487996b2833" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38397 } }
I20260928 03:09:47.485664 27764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:47.485786 27764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c01835430d94fa39248e487996b2833, State: Initialized, Role: FOLLOWER
I20260928 03:09:47.486573 27764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c01835430d94fa39248e487996b2833" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38397 } }
I20260928 03:09:47.486938 27764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:09:47.487087 27764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:09:47.487241 27764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:47.489578 27764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c01835430d94fa39248e487996b2833" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38397 } }
I20260928 03:09:47.490150 27764 leader_election.cc:302] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c01835430d94fa39248e487996b2833; no voters:
I20260928 03:09:47.490893 27764 leader_election.cc:288] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:09:47.491010 27774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:47.497483 27774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833 [term 1 LEADER]: Becoming Leader. State: Replica: 2c01835430d94fa39248e487996b2833, State: Running, Role: LEADER
I20260928 03:09:47.498036 27764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:09:47.498298 27774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c01835430d94fa39248e487996b2833" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38397 } }
I20260928 03:09:47.501533 27776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c01835430d94fa39248e487996b2833. Latest consensus state: current_term: 1 leader_uuid: "2c01835430d94fa39248e487996b2833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c01835430d94fa39248e487996b2833" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38397 } } }
I20260928 03:09:47.501524 27775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c01835430d94fa39248e487996b2833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c01835430d94fa39248e487996b2833" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38397 } } }
I20260928 03:09:47.502020 27775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833 [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:47.502020 27776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833 [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:47.504051 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:47.486541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36475 (local address 127.0.0.1:38397)
0928 03:09:47.486753 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:47.486760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:47.487292 (+ 532us) server_negotiation.cc:407] Connection header received
0928 03:09:47.487530 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:47.487535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:47.487610 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:47.487720 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:47.489149 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:47.490017 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:47.491829 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:47.492411 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:47.492495 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:47.495174 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:47.495180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:47.495184 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:47.495229 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:47.499143 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:47.500428 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:47.500438 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:47.500444 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:47.500768 (+ 324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:47.501442 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:47.501462 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:47.501469 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:47.502077 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:47.502481 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:47.503072 (+ 591us) server_negotiation.cc:299] Negotiation successful
0928 03:09:47.503422 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 03:09:47.508826 27780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:09:47.515195 27780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:09:47.525969 27780 catalog_manager.cc:1347] Generated new cluster ID: 3bb985ddf10c48cf8d0aac3303239469
I20260928 03:09:47.526100 27780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:09:47.546167 27780 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:09:47.547632 27780 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:09:47.563179 27780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c01835430d94fa39248e487996b2833: Generated new TSK 0
I20260928 03:09:47.564191 27780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:09:47.574011 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 27697
I20260928 03:09:47.575409 27759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:47.670715 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 27697
2026-09-28T03:09:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2185 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:09:47 dist-test-slave-fjww krb5kdc[27796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:09:47 dist-test-slave-fjww krb5kdc[27796](info): set up 2 sockets
Sep 28 03:09:47 dist-test-slave-fjww krb5kdc[27796](info): commencing operation
krb5kdc: starting...
W20260928 03:09:48.839983 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.005s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:09:49 dist-test-slave-fjww krb5kdc[27796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:09:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:09:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:09:49.127629 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:49.478798 27809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:49.479174 27809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:49.479355 27809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:49.487061 27809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:09:49.487246 27809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:49.487367 27809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:49.487512 27809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:09:49.487596 27809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:09:49.498665 27809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:49.500198 27809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:49.501541 27809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:49.514432 27814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:49.514432 27815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:49.514464 27817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:49.515250 27809 server_base.cc:1034] running on GCE node
I20260928 03:09:49.516158 27809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:49.517863 27809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:49.519119 27809 hybrid_clock.cc:648] HybridClock initialized: now 1790564989519048 us; error 58 us; skew 500 ppm
I20260928 03:09:49.519661 27809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:49 dist-test-slave-fjww krb5kdc[27796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:49.523211 27809 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:09:49.525024 27809 webserver.cc:460] Webserver started at http://127.0.0.1:44923/ using document root <none> and password file <none>
I20260928 03:09:49.525785 27809 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:49.525959 27809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:49.526279 27809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:49.529579 27809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "8899aca8e2f94d95ba5c578cc9941b97"
format_stamp: "Formatted at 2026-09-28 03:09:49 on dist-test-slave-fjww"
server_key: "4a26ef87987f37a785e523af033be99b"
server_key_iv: "ef331900104faca320e22b14559ff9a6"
server_key_version: "encryptionkey@0"
I20260928 03:09:49.530434 27809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "8899aca8e2f94d95ba5c578cc9941b97"
format_stamp: "Formatted at 2026-09-28 03:09:49 on dist-test-slave-fjww"
server_key: "4a26ef87987f37a785e523af033be99b"
server_key_iv: "ef331900104faca320e22b14559ff9a6"
server_key_version: "encryptionkey@0"
I20260928 03:09:49.536432 27809 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260928 03:09:49.540266 27824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:49.541723 27809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:09:49.541991 27809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "8899aca8e2f94d95ba5c578cc9941b97"
format_stamp: "Formatted at 2026-09-28 03:09:49 on dist-test-slave-fjww"
server_key: "4a26ef87987f37a785e523af033be99b"
server_key_iv: "ef331900104faca320e22b14559ff9a6"
server_key_version: "encryptionkey@0"
I20260928 03:09:49.542222 27809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:49.554700 27809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:49.559399 27809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:49.559767 27809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:49.579871 27875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40973 every 8 connection(s)
I20260928 03:09:49.579907 27809 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40973
I20260928 03:09:49.581573 27809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:09:49.585625 27876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:49.586021 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 27809
I20260928 03:09:49.586208 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:09:49.586683 6912 external_mini_cluster.cc:1428] Setting key 600cc5adb2551d8dafcf09852911c3b1
I20260928 03:09:49.600893 27876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97: Bootstrap starting.
I20260928 03:09:49.603730 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:49.588694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50869 (local address 127.0.0.1:40973)
0928 03:09:49.589519 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:49.589542 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:49.589592 (+ 50us) server_negotiation.cc:407] Connection header received
0928 03:09:49.592583 (+ 2991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:49.592617 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:49.593240 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:49.593739 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:49.594294 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:49.596192 (+ 1898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:49.597881 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:49.599033 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:49.599221 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:49.600578 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:49.600588 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:49.600610 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:49.601365 (+ 755us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:49.601477 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:49.601514 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:49.601634 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:49.601964 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 03:09:49.602258 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":318,"threads_started":1}
I20260928 03:09:49.605166 27876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:49.606387 27876 log.cc:824] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:49.608234 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:09:49.609756 27876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97: No bootstrap required, opened a new log
I20260928 03:09:49.619552 27876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8899aca8e2f94d95ba5c578cc9941b97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260928 03:09:49.620121 27876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:49.620193 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:49.612951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43669 (local address 127.0.0.1:40973)
0928 03:09:49.613147 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:49.613152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:49.613474 (+ 322us) server_negotiation.cc:407] Connection header received
0928 03:09:49.613641 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:49.613647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:49.613745 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:49.613882 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:49.614441 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:49.615633 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:49.617221 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:49.617994 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:49.618145 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:49.619323 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:49.619330 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:49.619334 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:49.619358 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:49.619409 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:49.619429 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:49.619562 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:49.619752 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 03:09:49.619854 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 03:09:49.620273 27876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899aca8e2f94d95ba5c578cc9941b97, State: Initialized, Role: FOLLOWER
I20260928 03:09:49.620971 27876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8899aca8e2f94d95ba5c578cc9941b97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260928 03:09:49.621265 27876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:09:49.621430 27876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:09:49.621615 27876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:49.623991 27876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8899aca8e2f94d95ba5c578cc9941b97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260928 03:09:49.624540 27876 leader_election.cc:302] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8899aca8e2f94d95ba5c578cc9941b97; no voters:
I20260928 03:09:49.625205 27876 leader_election.cc:288] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:09:49.625406 27886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:49.626168 27886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97 [term 1 LEADER]: Becoming Leader. State: Replica: 8899aca8e2f94d95ba5c578cc9941b97, State: Running, Role: LEADER
I20260928 03:09:49.626776 27886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8899aca8e2f94d95ba5c578cc9941b97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260928 03:09:49.627599 27876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:09:49.630196 27887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8899aca8e2f94d95ba5c578cc9941b97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8899aca8e2f94d95ba5c578cc9941b97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } } }
I20260928 03:09:49.630173 27888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8899aca8e2f94d95ba5c578cc9941b97. Latest consensus state: current_term: 1 leader_uuid: "8899aca8e2f94d95ba5c578cc9941b97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8899aca8e2f94d95ba5c578cc9941b97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } } }
I20260928 03:09:49.630671 27887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97 [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:49.630795 27888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97 [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:49.636421 27891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:09:49.641283 27891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:09:49.651095 27891 catalog_manager.cc:1347] Generated new cluster ID: 0e7cf2ac0ec84603ab45611204383062
I20260928 03:09:49.651228 27891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:09:49.674005 27891 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:09:49.675762 27891 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:09:49.683787 27891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8899aca8e2f94d95ba5c578cc9941b97: Generated new TSK 0
I20260928 03:09:49.684844 27891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:09:49.701349 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 27809
W20260928 03:09:49.702746 27871 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50869
W20260928 03:09:49.703791 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 27809: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:09:49.704054 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 27809
2026-09-28T03:09:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2034 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:09:49 dist-test-slave-fjww krb5kdc[27907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:09:49 dist-test-slave-fjww krb5kdc[27907](info): set up 2 sockets
Sep 28 03:09:49 dist-test-slave-fjww krb5kdc[27907](info): commencing operation
krb5kdc: starting...
W20260928 03:09:50.824235 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:09:51 dist-test-slave-fjww krb5kdc[27907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:09:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:09:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:09:51.116765 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44957
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:51.471695 27920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:51.472030 27920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:51.472241 27920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:51.480813 27920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:09:51.480949 27920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:51.481035 27920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:51.481143 27920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:09:51.481226 27920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:09:51.492491 27920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:51.494064 27920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:51.495621 27920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:51.509570 27925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:51.509730 27926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:51.509773 27928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:51.511191 27920 server_base.cc:1034] running on GCE node
I20260928 03:09:51.512192 27920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:51.514180 27920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:51.515558 27920 hybrid_clock.cc:648] HybridClock initialized: now 1790564991515478 us; error 65 us; skew 500 ppm
I20260928 03:09:51.516074 27920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:51 dist-test-slave-fjww krb5kdc[27907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:51.519560 27920 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:09:51.521256 27920 webserver.cc:460] Webserver started at http://127.0.0.1:41815/ using document root <none> and password file <none>
I20260928 03:09:51.522019 27920 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:51.522181 27920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:51.522509 27920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:51.525815 27920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "eb5272e656114a0aa63e3e6b4335c947"
format_stamp: "Formatted at 2026-09-28 03:09:51 on dist-test-slave-fjww"
server_key: "b1e0d56a0d9174ddf65239f1c27f8e6d"
server_key_iv: "c2d39aab0d43c38babb7ca4e2737f2b7"
server_key_version: "encryptionkey@0"
I20260928 03:09:51.526655 27920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "eb5272e656114a0aa63e3e6b4335c947"
format_stamp: "Formatted at 2026-09-28 03:09:51 on dist-test-slave-fjww"
server_key: "b1e0d56a0d9174ddf65239f1c27f8e6d"
server_key_iv: "c2d39aab0d43c38babb7ca4e2737f2b7"
server_key_version: "encryptionkey@0"
I20260928 03:09:51.532639 27920 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 03:09:51.536542 27935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:51.538055 27920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 03:09:51.538270 27920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "eb5272e656114a0aa63e3e6b4335c947"
format_stamp: "Formatted at 2026-09-28 03:09:51 on dist-test-slave-fjww"
server_key: "b1e0d56a0d9174ddf65239f1c27f8e6d"
server_key_iv: "c2d39aab0d43c38babb7ca4e2737f2b7"
server_key_version: "encryptionkey@0"
I20260928 03:09:51.538496 27920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:51.545197 27920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:51.549901 27920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:51.550231 27920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:51.571295 27920 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44957
I20260928 03:09:51.571313 27986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44957 every 8 connection(s)
I20260928 03:09:51.572845 27920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:09:51.575922 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 27920
I20260928 03:09:51.576108 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:09:51.576604 6912 external_mini_cluster.cc:1428] Setting key 9bcaff4027bb5ef7dc7813dbe855a447
I20260928 03:09:51.577299 27987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:51.586440 27990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:51.578570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53771 (local address 127.0.0.1:44957)
0928 03:09:51.579601 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:51.579619 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:51.579676 (+ 57us) server_negotiation.cc:407] Connection header received
0928 03:09:51.581802 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:51.581835 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:51.582389 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:51.582909 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:51.584079 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:51.584088 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:51.584114 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:51.584142 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:51.584232 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:51.584270 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:51.584527 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:51.584669 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 03:09:51.584936 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":431,"threads_started":1}
I20260928 03:09:51.591383 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:09:51.596302 27987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947: Bootstrap starting.
I20260928 03:09:51.602782 27987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:51.603845 27990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:51.599258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34263 (local address 127.0.0.1:44957)
0928 03:09:51.599541 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:51.599550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:51.601052 (+ 1502us) server_negotiation.cc:407] Connection header received
0928 03:09:51.601129 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:51.601134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:51.601206 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:51.601308 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:51.602620 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:51.602633 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:51.602638 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:51.602660 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:51.602720 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:51.602787 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:51.602922 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:51.603474 (+ 552us) server_negotiation.cc:299] Negotiation successful
0928 03:09:51.603540 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 03:09:51.604413 27987 log.cc:824] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:51.608677 27987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947: No bootstrap required, opened a new log
I20260928 03:09:51.618489 27987 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5272e656114a0aa63e3e6b4335c947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44957 } }
I20260928 03:09:51.618920 27987 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:51.619004 27987 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5272e656114a0aa63e3e6b4335c947, State: Initialized, Role: FOLLOWER
I20260928 03:09:51.619707 27987 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5272e656114a0aa63e3e6b4335c947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44957 } }
I20260928 03:09:51.619953 27987 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:09:51.620066 27987 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:09:51.620208 27987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:51.622754 27987 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5272e656114a0aa63e3e6b4335c947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44957 } }
I20260928 03:09:51.623364 27987 leader_election.cc:302] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb5272e656114a0aa63e3e6b4335c947; no voters:
I20260928 03:09:51.624214 27987 leader_election.cc:288] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:09:51.624378 27997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:51.625253 27997 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947 [term 1 LEADER]: Becoming Leader. State: Replica: eb5272e656114a0aa63e3e6b4335c947, State: Running, Role: LEADER
I20260928 03:09:51.626027 27997 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5272e656114a0aa63e3e6b4335c947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44957 } }
I20260928 03:09:51.626652 27987 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:09:51.629536 27999 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb5272e656114a0aa63e3e6b4335c947. Latest consensus state: current_term: 1 leader_uuid: "eb5272e656114a0aa63e3e6b4335c947" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5272e656114a0aa63e3e6b4335c947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44957 } } }
I20260928 03:09:51.629858 27998 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb5272e656114a0aa63e3e6b4335c947" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5272e656114a0aa63e3e6b4335c947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44957 } } }
I20260928 03:09:51.630143 27999 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947 [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:51.630187 27998 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947 [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:51.632033 28007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:09:51.637842 28007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:09:51.647921 28007 catalog_manager.cc:1347] Generated new cluster ID: f2e8da4216e448b0976a423e8fdc9ef6
I20260928 03:09:51.648072 28007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:09:51.667755 28007 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:09:51.668329 28007 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:09:51.684748 28007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb5272e656114a0aa63e3e6b4335c947: Generated new TSK 0
I20260928 03:09:51.685793 28007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:09:51.703523 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 27920
W20260928 03:09:51.704916 27982 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53771
W20260928 03:09:51.705726 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 27920: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:09:51.705869 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 27920
2026-09-28T03:09:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2000 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:09:51 dist-test-slave-fjww krb5kdc[28018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:09:51 dist-test-slave-fjww krb5kdc[28018](info): set up 2 sockets
Sep 28 03:09:51 dist-test-slave-fjww krb5kdc[28018](info): commencing operation
krb5kdc: starting...
W20260928 03:09:52.876202 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:09:53 dist-test-slave-fjww krb5kdc[28018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:09:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:09:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:09:53.168820 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:53.522863 28031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:53.523262 28031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:53.523494 28031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:53.531872 28031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:09:53.532089 28031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:53.532199 28031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:53.532322 28031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:09:53.532410 28031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:09:53.543342 28031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:53.544787 28031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:53.546150 28031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:53.559309 28039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:53.559355 28037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:53.559417 28036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:53.560488 28031 server_base.cc:1034] running on GCE node
I20260928 03:09:53.561386 28031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:53.563082 28031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:53.564361 28031 hybrid_clock.cc:648] HybridClock initialized: now 1790564993564303 us; error 62 us; skew 500 ppm
I20260928 03:09:53.564782 28031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:53 dist-test-slave-fjww krb5kdc[28018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:53.573254 28031 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:09:53.574924 28031 webserver.cc:460] Webserver started at http://127.0.0.1:35985/ using document root <none> and password file <none>
I20260928 03:09:53.575798 28031 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:53.575919 28031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:53.576196 28031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:53.579309 28031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "b0e4c44123d54d6f8660c68f50e0d5fb"
format_stamp: "Formatted at 2026-09-28 03:09:53 on dist-test-slave-fjww"
server_key: "ef45b73de6e44537c8228492a078eeaa"
server_key_iv: "9efc5dc704d874e6299f2dc48d36b37f"
server_key_version: "encryptionkey@0"
I20260928 03:09:53.580156 28031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "b0e4c44123d54d6f8660c68f50e0d5fb"
format_stamp: "Formatted at 2026-09-28 03:09:53 on dist-test-slave-fjww"
server_key: "ef45b73de6e44537c8228492a078eeaa"
server_key_iv: "9efc5dc704d874e6299f2dc48d36b37f"
server_key_version: "encryptionkey@0"
I20260928 03:09:53.585613 28031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 03:09:53.589418 28046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:53.590911 28031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 03:09:53.591110 28031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "b0e4c44123d54d6f8660c68f50e0d5fb"
format_stamp: "Formatted at 2026-09-28 03:09:53 on dist-test-slave-fjww"
server_key: "ef45b73de6e44537c8228492a078eeaa"
server_key_iv: "9efc5dc704d874e6299f2dc48d36b37f"
server_key_version: "encryptionkey@0"
I20260928 03:09:53.591347 28031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:53.610014 28031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:53.614861 28031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:53.615200 28031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:53.635592 28031 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32969
I20260928 03:09:53.635718 28097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32969 every 8 connection(s)
I20260928 03:09:53.637120 28031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:09:53.641028 28098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:53.646880 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 28031
I20260928 03:09:53.647099 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:09:53.647637 6912 external_mini_cluster.cc:1428] Setting key c56f9d17ccce6f1de208aeb88a52c480
I20260928 03:09:53.652339 28098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb: Bootstrap starting.
I20260928 03:09:53.656329 28098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:53.657667 28098 log.cc:824] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:53.660939 28098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb: No bootstrap required, opened a new log
Sep 28 03:09:53 dist-test-slave-fjww krb5kdc[28018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 03:09:53.669660 28098 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e4c44123d54d6f8660c68f50e0d5fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260928 03:09:53.670114 28098 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:53.670262 28098 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e4c44123d54d6f8660c68f50e0d5fb, State: Initialized, Role: FOLLOWER
I20260928 03:09:53.670985 28098 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e4c44123d54d6f8660c68f50e0d5fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260928 03:09:53.671284 28098 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:09:53.671496 28098 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:09:53.671708 28098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:53.672874 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:53.649695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:44177 (local address 127.0.0.1:32969)
0928 03:09:53.650570 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:53.650582 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:53.650623 (+ 41us) server_negotiation.cc:407] Connection header received
0928 03:09:53.652564 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:53.652591 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:53.653023 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:53.653346 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:53.655288 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:53.657046 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:53.658856 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:53.659641 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:53.659852 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:53.664496 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:53.664504 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:53.664525 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:53.664570 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:53.668006 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:53.668934 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:53.668942 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:53.668950 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:53.669123 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:53.669599 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:53.669604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:53.669608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:53.670362 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:53.670745 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:53.671272 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 03:09:53.671718 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":409,"threads_started":1}
I20260928 03:09:53.674320 28098 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e4c44123d54d6f8660c68f50e0d5fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260928 03:09:53.674952 28098 leader_election.cc:302] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0e4c44123d54d6f8660c68f50e0d5fb; no voters:
I20260928 03:09:53.675666 28098 leader_election.cc:288] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:09:53.675796 28103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:53.677461 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:09:53.677446 28103 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb [term 1 LEADER]: Becoming Leader. State: Replica: b0e4c44123d54d6f8660c68f50e0d5fb, State: Running, Role: LEADER
I20260928 03:09:53.678076 28103 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e4c44123d54d6f8660c68f50e0d5fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260928 03:09:53.678401 28098 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:09:53.688069 28104 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0e4c44123d54d6f8660c68f50e0d5fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e4c44123d54d6f8660c68f50e0d5fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } } }
I20260928 03:09:53.688594 28104 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:53.689379 28105 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0e4c44123d54d6f8660c68f50e0d5fb. Latest consensus state: current_term: 1 leader_uuid: "b0e4c44123d54d6f8660c68f50e0d5fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e4c44123d54d6f8660c68f50e0d5fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } } }
I20260928 03:09:53.689743 28105 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:53.692900 28114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:09:53.700599 28114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:09:53.702730 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:53.682540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:47193 (local address 127.0.0.1:32969)
0928 03:09:53.682732 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:53.682740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:53.684056 (+ 1316us) server_negotiation.cc:407] Connection header received
0928 03:09:53.684268 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:53.684273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:53.684353 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:53.684467 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:53.686018 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:53.687708 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:53.689427 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:53.690016 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:53.690161 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:53.693697 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:53.693702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:53.693706 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:53.693740 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:53.697844 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:53.699105 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:53.699111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:53.699114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:53.699262 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:53.699938 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:53.699943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:53.699948 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:53.700411 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:53.701891 (+ 1480us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:53.702093 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 03:09:53.702322 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 03:09:53.713263 28114 catalog_manager.cc:1347] Generated new cluster ID: 7c1f33240577451cb354c75b68c3628a
I20260928 03:09:53.713403 28114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:09:53.747269 28114 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:09:53.748813 28114 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:09:53.760886 28114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0e4c44123d54d6f8660c68f50e0d5fb: Generated new TSK 0
I20260928 03:09:53.761936 28114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:09:53.769140 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 28031
I20260928 03:09:53.770840 28093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:09:53.871327 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 28031
W20260928 03:09:53.885836 28016 connection.cc:535] client connection to 127.0.0.1:32969 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32969): Connection reset by peer (error 104)
2026-09-28T03:09:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2161 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:09:53 dist-test-slave-fjww krb5kdc[28130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:09:53 dist-test-slave-fjww krb5kdc[28130](info): set up 2 sockets
Sep 28 03:09:53 dist-test-slave-fjww krb5kdc[28130](info): commencing operation
krb5kdc: starting...
W20260928 03:09:55.012221 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:09:55 dist-test-slave-fjww krb5kdc[28130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:09:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:09:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:09:55.292919 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40515
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:55.645193 28143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:55.645506 28143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:55.645686 28143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:55.653262 28143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:09:55.653399 28143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:55.653517 28143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:55.653630 28143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:09:55.653697 28143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:09:55.665850 28143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:55.667379 28143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:55.668987 28143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:55.682571 28148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:55.682806 28149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:55.682852 28151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:55.683547 28143 server_base.cc:1034] running on GCE node
I20260928 03:09:55.684404 28143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:55.685914 28143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:55.687206 28143 hybrid_clock.cc:648] HybridClock initialized: now 1790564995687151 us; error 51 us; skew 500 ppm
I20260928 03:09:55.687729 28143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:55 dist-test-slave-fjww krb5kdc[28130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:55.691105 28143 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:09:55.692850 28143 webserver.cc:460] Webserver started at http://127.0.0.1:43329/ using document root <none> and password file <none>
I20260928 03:09:55.693596 28143 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:55.693753 28143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:55.694073 28143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:55.697273 28143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "133883d463844c16a0017693186ce40c"
format_stamp: "Formatted at 2026-09-28 03:09:55 on dist-test-slave-fjww"
server_key: "b4ad9eef9cbfbd9676f2f79aab7c91c6"
server_key_iv: "b3487092931e27084df52944a48ead3b"
server_key_version: "encryptionkey@0"
I20260928 03:09:55.698076 28143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "133883d463844c16a0017693186ce40c"
format_stamp: "Formatted at 2026-09-28 03:09:55 on dist-test-slave-fjww"
server_key: "b4ad9eef9cbfbd9676f2f79aab7c91c6"
server_key_iv: "b3487092931e27084df52944a48ead3b"
server_key_version: "encryptionkey@0"
I20260928 03:09:55.703830 28143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 03:09:55.708086 28158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:55.709694 28143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 03:09:55.709911 28143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "133883d463844c16a0017693186ce40c"
format_stamp: "Formatted at 2026-09-28 03:09:55 on dist-test-slave-fjww"
server_key: "b4ad9eef9cbfbd9676f2f79aab7c91c6"
server_key_iv: "b3487092931e27084df52944a48ead3b"
server_key_version: "encryptionkey@0"
I20260928 03:09:55.710153 28143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:55.726775 28143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:55.731663 28143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:55.732004 28143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:55.752367 28143 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40515
I20260928 03:09:55.752401 28209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40515 every 8 connection(s)
I20260928 03:09:55.753949 28143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:09:55.757818 28210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:55.760896 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 28143
I20260928 03:09:55.761135 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:09:55.761600 6912 external_mini_cluster.cc:1428] Setting key 9e87b4c5b69597bc5cd8ddb08156bbec
I20260928 03:09:55.772083 28210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c: Bootstrap starting.
I20260928 03:09:55.776156 28210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:55.777603 28210 log.cc:824] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:55.778024 28213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:55.763679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:40395 (local address 127.0.0.1:40515)
0928 03:09:55.764508 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:55.764523 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:55.764572 (+ 49us) server_negotiation.cc:407] Connection header received
0928 03:09:55.766943 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:55.766976 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:55.767733 (+ 757us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:55.768224 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:55.768916 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:55.770990 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:55.772581 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:55.773597 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:55.773815 (+ 218us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:55.775008 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:55.775019 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:55.775038 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:55.775779 (+ 741us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:55.775890 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:55.775926 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:55.776134 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:55.776422 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 03:09:55.776621 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":326,"threads_started":1}
I20260928 03:09:55.781002 28210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c: No bootstrap required, opened a new log
I20260928 03:09:55.782655 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:09:55.789860 28210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133883d463844c16a0017693186ce40c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40515 } }
I20260928 03:09:55.790400 28210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:55.790525 28210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133883d463844c16a0017693186ce40c, State: Initialized, Role: FOLLOWER
I20260928 03:09:55.791157 28210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133883d463844c16a0017693186ce40c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40515 } }
I20260928 03:09:55.791426 28210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:09:55.791690 28210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:09:55.791887 28210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:55.794248 28210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133883d463844c16a0017693186ce40c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40515 } }
I20260928 03:09:55.794669 28213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:55.787425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:36337 (local address 127.0.0.1:40515)
0928 03:09:55.787646 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:55.787652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:55.788001 (+ 349us) server_negotiation.cc:407] Connection header received
0928 03:09:55.788162 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:55.788168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:55.788259 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:55.788393 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:55.788926 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:55.790030 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:55.792018 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:55.792472 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:55.792633 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:55.793845 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:55.793850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:55.793853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:55.793873 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:55.793930 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:55.793945 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:55.794072 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:55.794282 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 03:09:55.794340 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 03:09:55.795008 28210 leader_election.cc:302] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 133883d463844c16a0017693186ce40c; no voters:
I20260928 03:09:55.795781 28210 leader_election.cc:288] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:09:55.795900 28220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:55.796619 28220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c [term 1 LEADER]: Becoming Leader. State: Replica: 133883d463844c16a0017693186ce40c, State: Running, Role: LEADER
I20260928 03:09:55.797372 28220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133883d463844c16a0017693186ce40c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40515 } }
I20260928 03:09:55.799988 28221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 133883d463844c16a0017693186ce40c. Latest consensus state: current_term: 1 leader_uuid: "133883d463844c16a0017693186ce40c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133883d463844c16a0017693186ce40c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40515 } } }
I20260928 03:09:55.800479 28221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:55.800654 28222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "133883d463844c16a0017693186ce40c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133883d463844c16a0017693186ce40c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40515 } } }
I20260928 03:09:55.801064 28222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:55.801664 28210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:09:55.805310 28224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:09:55.810211 28224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:09:55.820613 28224 catalog_manager.cc:1347] Generated new cluster ID: 9f277f9d934348c193e74034bda747d9
I20260928 03:09:55.820768 28224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:09:55.831856 28224 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:09:55.833303 28224 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:09:55.845578 28224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 133883d463844c16a0017693186ce40c: Generated new TSK 0
I20260928 03:09:55.846524 28224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:09:55.854050 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 28143
W20260928 03:09:55.855391 28205 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.198:40395
W20260928 03:09:55.856364 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 28143: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:09:55.856536 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 28143
W20260928 03:09:55.870611 28128 connection.cc:535] client connection to 127.0.0.1:40515 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40515): Connection reset by peer (error 104)
2026-09-28T03:09:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1984 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:09:55 dist-test-slave-fjww krb5kdc[28241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:09:55 dist-test-slave-fjww krb5kdc[28241](info): set up 2 sockets
Sep 28 03:09:55 dist-test-slave-fjww krb5kdc[28241](info): commencing operation
krb5kdc: starting...
W20260928 03:09:56.996724 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:09:57 dist-test-slave-fjww krb5kdc[28241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:09:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:09:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:09:57.280405 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:57.635944 28254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:57.636278 28254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:57.636441 28254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:57.644140 28254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:09:57.644284 28254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:57.644407 28254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:57.644518 28254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:09:57.644601 28254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:09:57.655529 28254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:57.657022 28254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:57.658404 28254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:57.671980 28259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:57.672071 28260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:57.672206 28262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:57.672757 28254 server_base.cc:1034] running on GCE node
I20260928 03:09:57.673666 28254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:57.675207 28254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:57.676419 28254 hybrid_clock.cc:648] HybridClock initialized: now 1790564997676349 us; error 69 us; skew 500 ppm
I20260928 03:09:57.676852 28254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:57 dist-test-slave-fjww krb5kdc[28241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:57.680406 28254 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:09:57.682188 28254 webserver.cc:460] Webserver started at http://127.0.0.1:39227/ using document root <none> and password file <none>
I20260928 03:09:57.682984 28254 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:57.683140 28254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:57.683516 28254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:57.686637 28254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "dc16beb1b9f44b1ebbb60fc3070355ea"
format_stamp: "Formatted at 2026-09-28 03:09:57 on dist-test-slave-fjww"
server_key: "0ac6f5643015ddd22b78268da2cd0c6f"
server_key_iv: "50320bddfe3298d0ac7bc37c0a4c6adb"
server_key_version: "encryptionkey@0"
I20260928 03:09:57.687577 28254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "dc16beb1b9f44b1ebbb60fc3070355ea"
format_stamp: "Formatted at 2026-09-28 03:09:57 on dist-test-slave-fjww"
server_key: "0ac6f5643015ddd22b78268da2cd0c6f"
server_key_iv: "50320bddfe3298d0ac7bc37c0a4c6adb"
server_key_version: "encryptionkey@0"
I20260928 03:09:57.694031 28254 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 03:09:57.698392 28269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:57.699934 28254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 03:09:57.700156 28254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "dc16beb1b9f44b1ebbb60fc3070355ea"
format_stamp: "Formatted at 2026-09-28 03:09:57 on dist-test-slave-fjww"
server_key: "0ac6f5643015ddd22b78268da2cd0c6f"
server_key_iv: "50320bddfe3298d0ac7bc37c0a4c6adb"
server_key_version: "encryptionkey@0"
I20260928 03:09:57.700384 28254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:57.706660 28254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:57.711630 28254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:57.711968 28254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:57.732935 28254 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34113
I20260928 03:09:57.732992 28320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34113 every 8 connection(s)
I20260928 03:09:57.734519 28254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:09:57.738257 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 28254
I20260928 03:09:57.738467 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:09:57.738936 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:09:57.738952 6912 external_mini_cluster.cc:1428] Setting key 20ecdf4e1a3ff7f801520ca788e72645
I20260928 03:09:57.747897 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:57.740913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:38403 (local address 127.0.0.1:34113)
0928 03:09:57.741811 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:57.741822 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:57.741866 (+ 44us) server_negotiation.cc:407] Connection header received
0928 03:09:57.743842 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:57.743874 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:57.744273 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:57.744631 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:57.746013 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:57.746019 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:57.746033 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:57.746052 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:57.746117 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:57.746149 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:57.746337 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:57.746513 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 03:09:57.746760 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":426,"threads_started":1}
I20260928 03:09:57.752020 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:09:57.754148 28321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea: Bootstrap starting.
I20260928 03:09:57.758191 28321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:57.759565 28321 log.cc:824] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:57.759984 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:57.756957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:35063 (local address 127.0.0.1:34113)
0928 03:09:57.757211 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:57.757221 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:57.757749 (+ 528us) server_negotiation.cc:407] Connection header received
0928 03:09:57.757978 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:57.757983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:57.758044 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:57.758164 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:57.759189 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:57.759195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:57.759199 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:09:57.759221 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:57.759271 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:09:57.759287 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:57.759425 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:57.759553 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 03:09:57.759630 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 03:09:57.763044 28321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea: No bootstrap required, opened a new log
I20260928 03:09:57.771677 28321 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc16beb1b9f44b1ebbb60fc3070355ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34113 } }
I20260928 03:09:57.772158 28321 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:57.772274 28321 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc16beb1b9f44b1ebbb60fc3070355ea, State: Initialized, Role: FOLLOWER
I20260928 03:09:57.772949 28321 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc16beb1b9f44b1ebbb60fc3070355ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34113 } }
I20260928 03:09:57.773213 28321 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:09:57.773386 28321 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:09:57.773559 28321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:57.776017 28321 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc16beb1b9f44b1ebbb60fc3070355ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34113 } }
I20260928 03:09:57.776616 28321 leader_election.cc:302] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc16beb1b9f44b1ebbb60fc3070355ea; no voters:
I20260928 03:09:57.777346 28321 leader_election.cc:288] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:09:57.777486 28331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:57.778419 28331 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea [term 1 LEADER]: Becoming Leader. State: Replica: dc16beb1b9f44b1ebbb60fc3070355ea, State: Running, Role: LEADER
I20260928 03:09:57.779099 28331 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc16beb1b9f44b1ebbb60fc3070355ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34113 } }
I20260928 03:09:57.779982 28321 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:09:57.782903 28332 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc16beb1b9f44b1ebbb60fc3070355ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc16beb1b9f44b1ebbb60fc3070355ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34113 } } }
I20260928 03:09:57.783329 28332 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:57.784260 28333 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc16beb1b9f44b1ebbb60fc3070355ea. Latest consensus state: current_term: 1 leader_uuid: "dc16beb1b9f44b1ebbb60fc3070355ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc16beb1b9f44b1ebbb60fc3070355ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34113 } } }
I20260928 03:09:57.784782 28333 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:57.786764 28338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:09:57.792550 28338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:09:57.802816 28338 catalog_manager.cc:1347] Generated new cluster ID: 15ad13523c184c10865be7df534aa717
I20260928 03:09:57.803033 28338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:09:57.820156 28338 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:09:57.820797 28338 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:09:57.834213 28338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc16beb1b9f44b1ebbb60fc3070355ea: Generated new TSK 0
I20260928 03:09:57.835403 28338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:09:57.841238 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 28254
W20260928 03:09:57.842571 28316 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.198:38403
W20260928 03:09:57.843433 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 28254: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:09:57.843679 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 28254
2026-09-28T03:09:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1986 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:09:57 dist-test-slave-fjww krb5kdc[28352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:09:57 dist-test-slave-fjww krb5kdc[28352](info): set up 2 sockets
Sep 28 03:09:57 dist-test-slave-fjww krb5kdc[28352](info): commencing operation
krb5kdc: starting...
W20260928 03:09:59.072192 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.004s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:09:59 dist-test-slave-fjww krb5kdc[28352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:09:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:09:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:09:59.362924 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45681
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:09:59.727169 28365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:09:59.727617 28365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:09:59.727783 28365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:09:59.735764 28365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:09:59.735895 28365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:09:59.736003 28365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:09:59.736112 28365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:09:59.736181 28365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:09:59.746672 28365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45681
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:09:59.748200 28365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:09:59.749596 28365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:09:59.763141 28370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:59.763090 28371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:09:59.763062 28373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:09:59.764276 28365 server_base.cc:1034] running on GCE node
I20260928 03:09:59.765282 28365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:09:59.766837 28365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:09:59.768112 28365 hybrid_clock.cc:648] HybridClock initialized: now 1790564999768051 us; error 46 us; skew 500 ppm
I20260928 03:09:59.768596 28365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:09:59 dist-test-slave-fjww krb5kdc[28352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:09:59.776458 28365 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:09:59.778236 28365 webserver.cc:460] Webserver started at http://127.0.0.1:36477/ using document root <none> and password file <none>
I20260928 03:09:59.778975 28365 fs_manager.cc:359] Metadata directory not provided
I20260928 03:09:59.779084 28365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:09:59.779364 28365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:09:59.782595 28365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "f5d8c644ff2243719fe0a98037b9aefb"
format_stamp: "Formatted at 2026-09-28 03:09:59 on dist-test-slave-fjww"
server_key: "1867ccc54425602ea4d7de063a1381a6"
server_key_iv: "cc23ae23272d5dfd8dc1fd755cf7b7c3"
server_key_version: "encryptionkey@0"
I20260928 03:09:59.783344 28365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "f5d8c644ff2243719fe0a98037b9aefb"
format_stamp: "Formatted at 2026-09-28 03:09:59 on dist-test-slave-fjww"
server_key: "1867ccc54425602ea4d7de063a1381a6"
server_key_iv: "cc23ae23272d5dfd8dc1fd755cf7b7c3"
server_key_version: "encryptionkey@0"
I20260928 03:09:59.789047 28365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 03:09:59.793373 28380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:09:59.795061 28365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 03:09:59.795238 28365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "f5d8c644ff2243719fe0a98037b9aefb"
format_stamp: "Formatted at 2026-09-28 03:09:59 on dist-test-slave-fjww"
server_key: "1867ccc54425602ea4d7de063a1381a6"
server_key_iv: "cc23ae23272d5dfd8dc1fd755cf7b7c3"
server_key_version: "encryptionkey@0"
I20260928 03:09:59.795598 28365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:09:59.814008 28365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:09:59.818490 28365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:09:59.818751 28365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:09:59.838809 28365 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45681
I20260928 03:09:59.838842 28431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45681 every 8 connection(s)
I20260928 03:09:59.840339 28365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:09:59.841885 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 28365
I20260928 03:09:59.842082 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:09:59.842644 6912 external_mini_cluster.cc:1428] Setting key 324de6ef6e0f4a048efdf42c1039ab8c
I20260928 03:09:59.845278 28432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 03:09:59 dist-test-slave-fjww krb5kdc[28352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 03:09:59.859387 28432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb: Bootstrap starting.
I20260928 03:09:59.863183 28432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb: Neither blocks nor log segments found. Creating new log.
I20260928 03:09:59.864588 28432 log.cc:824] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb: Log is configured to *not* fsync() on all Append() calls
I20260928 03:09:59.868475 28432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb: No bootstrap required, opened a new log
I20260928 03:09:59.869649 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:59.844744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:41949 (local address 127.0.0.1:45681)
0928 03:09:59.845651 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:59.845664 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:59.845707 (+ 43us) server_negotiation.cc:407] Connection header received
0928 03:09:59.847425 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:59.847514 (+ 89us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:59.847958 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:59.848320 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:59.850160 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:59.851835 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:59.853649 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:59.854346 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:59.854499 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:59.860027 (+ 5528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:59.860037 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:59.860056 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:59.860104 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:59.864102 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:59.865310 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:59.865320 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:59.865329 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:59.865522 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:59.866151 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:59.866156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:59.866159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:59.866851 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:59.867206 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:59.867801 (+ 595us) server_negotiation.cc:299] Negotiation successful
0928 03:09:59.868226 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":428,"threads_started":1}
I20260928 03:09:59.874359 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:09:59.878674 28432 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d8c644ff2243719fe0a98037b9aefb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45681 } }
I20260928 03:09:59.879200 28432 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:09:59.879311 28432 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d8c644ff2243719fe0a98037b9aefb, State: Initialized, Role: FOLLOWER
I20260928 03:09:59.880136 28432 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d8c644ff2243719fe0a98037b9aefb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45681 } }
I20260928 03:09:59.880499 28432 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:09:59.880651 28432 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:09:59.880852 28432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:09:59.884099 28432 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d8c644ff2243719fe0a98037b9aefb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45681 } }
I20260928 03:09:59.884755 28432 leader_election.cc:302] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5d8c644ff2243719fe0a98037b9aefb; no voters:
I20260928 03:09:59.885455 28432 leader_election.cc:288] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:09:59.885586 28442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:09:59.887156 28442 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb [term 1 LEADER]: Becoming Leader. State: Replica: f5d8c644ff2243719fe0a98037b9aefb, State: Running, Role: LEADER
I20260928 03:09:59.887874 28442 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d8c644ff2243719fe0a98037b9aefb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45681 } }
I20260928 03:09:59.888207 28432 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:09:59.891474 28443 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5d8c644ff2243719fe0a98037b9aefb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d8c644ff2243719fe0a98037b9aefb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45681 } } }
I20260928 03:09:59.891901 28443 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:59.891824 28444 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5d8c644ff2243719fe0a98037b9aefb. Latest consensus state: current_term: 1 leader_uuid: "f5d8c644ff2243719fe0a98037b9aefb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d8c644ff2243719fe0a98037b9aefb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45681 } } }
I20260928 03:09:59.892251 28444 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb [sys.catalog]: This master's current role is: LEADER
I20260928 03:09:59.893381 28448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:09:59.900315 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:09:59.880047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:43359 (local address 127.0.0.1:45681)
0928 03:09:59.880286 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 03:09:59.880296 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:09:59.880720 (+ 424us) server_negotiation.cc:407] Connection header received
0928 03:09:59.880956 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:09:59.880963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:09:59.881057 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:09:59.881141 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:09:59.883037 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:59.883951 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:59.885831 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:09:59.886585 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:09:59.886679 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:09:59.890523 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:09:59.890530 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:09:59.890535 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:09:59.890578 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:09:59.895127 (+ 4549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:59.896576 (+ 1449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:59.896586 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:59.896592 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:59.896923 (+ 331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:09:59.897705 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:09:59.897713 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:09:59.897718 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:09:59.898235 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:09:59.898641 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0928 03:09:59.899616 (+ 975us) server_negotiation.cc:299] Negotiation successful
0928 03:09:59.899886 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 03:09:59.903929 28448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:09:59.918776 28448 catalog_manager.cc:1347] Generated new cluster ID: 52427ded9c434a24bf405e0dafe12de2
I20260928 03:09:59.918901 28448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:09:59.931367 28448 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:09:59.932797 28448 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:09:59.944139 28448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5d8c644ff2243719fe0a98037b9aefb: Generated new TSK 0
I20260928 03:09:59.945062 28448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:09:59.956869 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 28365
I20260928 03:09:59.958328 28427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:10:00.052182 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 28365
W20260928 03:10:00.071762 28350 connection.cc:535] client connection to 127.0.0.1:45681 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45681): Connection reset by peer (error 104)
2026-09-28T03:10:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2219 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:10:00 dist-test-slave-fjww krb5kdc[28464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:10:00 dist-test-slave-fjww krb5kdc[28464](info): set up 2 sockets
Sep 28 03:10:00 dist-test-slave-fjww krb5kdc[28464](info): commencing operation
krb5kdc: starting...
W20260928 03:10:01.216650 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:10:01 dist-test-slave-fjww krb5kdc[28464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:10:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:10:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:10:01.504125 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40895
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:10:01.861888 28477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:10:01.862232 28477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:10:01.862408 28477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:10:01.870060 28477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:10:01.870201 28477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:10:01.870303 28477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:10:01.870425 28477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:10:01.870499 28477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:10:01.881476 28477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40895
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:10:01.882913 28477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:10:01.884413 28477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:10:01.897400 28482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:01.897476 28483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:01.897509 28485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:10:01.898416 28477 server_base.cc:1034] running on GCE node
I20260928 03:10:01.899312 28477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:10:01.900954 28477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:10:01.902182 28477 hybrid_clock.cc:648] HybridClock initialized: now 1790565001902099 us; error 79 us; skew 500 ppm
I20260928 03:10:01.902626 28477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:10:01 dist-test-slave-fjww krb5kdc[28464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:10:01.906196 28477 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:10:01.907923 28477 webserver.cc:460] Webserver started at http://127.0.0.1:36259/ using document root <none> and password file <none>
I20260928 03:10:01.908607 28477 fs_manager.cc:359] Metadata directory not provided
I20260928 03:10:01.908715 28477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:10:01.909062 28477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:10:01.912362 28477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "a192759ae7564e6da73d469c7ff11cab"
format_stamp: "Formatted at 2026-09-28 03:10:01 on dist-test-slave-fjww"
server_key: "cc85b7452269705955d52fa74ba613cb"
server_key_iv: "20af00e6dff893369e4f3b03f2cdcab7"
server_key_version: "encryptionkey@0"
I20260928 03:10:01.913147 28477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "a192759ae7564e6da73d469c7ff11cab"
format_stamp: "Formatted at 2026-09-28 03:10:01 on dist-test-slave-fjww"
server_key: "cc85b7452269705955d52fa74ba613cb"
server_key_iv: "20af00e6dff893369e4f3b03f2cdcab7"
server_key_version: "encryptionkey@0"
I20260928 03:10:01.918638 28477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 03:10:01.922797 28492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:10:01.924579 28477 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:10:01.924813 28477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "a192759ae7564e6da73d469c7ff11cab"
format_stamp: "Formatted at 2026-09-28 03:10:01 on dist-test-slave-fjww"
server_key: "cc85b7452269705955d52fa74ba613cb"
server_key_iv: "20af00e6dff893369e4f3b03f2cdcab7"
server_key_version: "encryptionkey@0"
I20260928 03:10:01.925064 28477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:10:01.949184 28477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:10:01.954332 28477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:10:01.954681 28477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:10:01.976225 28477 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40895
I20260928 03:10:01.976282 28543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40895 every 8 connection(s)
I20260928 03:10:01.977725 28477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:10:01.982277 28544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:10:01.982884 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 28477
I20260928 03:10:01.983103 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:10:01.983791 6912 external_mini_cluster.cc:1428] Setting key e6af9d6f08435a737fff058d618c39e1
I20260928 03:10:01.997951 28544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab: Bootstrap starting.
I20260928 03:10:02.000598 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:01.986013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:52391 (local address 127.0.0.1:40895)
0928 03:10:01.986996 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:01.987010 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:01.987057 (+ 47us) server_negotiation.cc:407] Connection header received
0928 03:10:01.989915 (+ 2858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:01.989949 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:01.990521 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:01.991017 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:01.991715 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:01.993205 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:01.994851 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:01.995861 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:01.996101 (+ 240us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:01.997671 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:01.997679 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:01.997697 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:10:01.998400 (+ 703us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:01.998496 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:10:01.998538 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:01.998731 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:01.999249 (+ 518us) server_negotiation.cc:299] Negotiation successful
0928 03:10:01.999523 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":431,"threads_started":1}
I20260928 03:10:02.001899 28544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab: Neither blocks nor log segments found. Creating new log.
I20260928 03:10:02.003288 28544 log.cc:824] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab: Log is configured to *not* fsync() on all Append() calls
I20260928 03:10:02.005159 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:10:02.006769 28544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab: No bootstrap required, opened a new log
I20260928 03:10:02.017356 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:02.009931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:38555 (local address 127.0.0.1:40895)
0928 03:10:02.010130 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:02.010136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:02.010673 (+ 537us) server_negotiation.cc:407] Connection header received
0928 03:10:02.010910 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:02.010914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:02.010985 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:02.011066 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:02.011717 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:02.012526 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:02.014494 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:02.015002 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:02.015114 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:02.016470 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:02.016475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:02.016479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:10:02.016498 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:02.016555 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:10:02.016572 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:02.016667 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:02.016909 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 03:10:02.016956 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 03:10:02.017263 28544 raft_consensus.cc:348] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a192759ae7564e6da73d469c7ff11cab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40895 } }
I20260928 03:10:02.017838 28544 raft_consensus.cc:374] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:10:02.017971 28544 raft_consensus.cc:729] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a192759ae7564e6da73d469c7ff11cab, State: Initialized, Role: FOLLOWER
I20260928 03:10:02.018589 28544 consensus_queue.cc:260] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a192759ae7564e6da73d469c7ff11cab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40895 } }
I20260928 03:10:02.018873 28544 raft_consensus.cc:388] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:10:02.019073 28544 raft_consensus.cc:482] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:10:02.019273 28544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:10:02.021855 28544 raft_consensus.cc:504] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a192759ae7564e6da73d469c7ff11cab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40895 } }
I20260928 03:10:02.022543 28544 leader_election.cc:302] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a192759ae7564e6da73d469c7ff11cab; no voters:
I20260928 03:10:02.023216 28544 leader_election.cc:288] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:10:02.023380 28554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:10:02.024385 28554 raft_consensus.cc:686] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab [term 1 LEADER]: Becoming Leader. State: Replica: a192759ae7564e6da73d469c7ff11cab, State: Running, Role: LEADER
I20260928 03:10:02.025066 28554 consensus_queue.cc:237] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a192759ae7564e6da73d469c7ff11cab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40895 } }
I20260928 03:10:02.025586 28544 sys_catalog.cc:564] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:10:02.027992 28555 sys_catalog.cc:455] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a192759ae7564e6da73d469c7ff11cab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a192759ae7564e6da73d469c7ff11cab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40895 } } }
I20260928 03:10:02.028585 28555 sys_catalog.cc:458] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:02.029197 28556 sys_catalog.cc:455] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab [sys.catalog]: SysCatalogTable state changed. Reason: New leader a192759ae7564e6da73d469c7ff11cab. Latest consensus state: current_term: 1 leader_uuid: "a192759ae7564e6da73d469c7ff11cab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a192759ae7564e6da73d469c7ff11cab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40895 } } }
I20260928 03:10:02.029510 28556 sys_catalog.cc:458] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:02.036628 28560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:10:02.042418 28560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:10:02.053246 28560 catalog_manager.cc:1347] Generated new cluster ID: 0368c794bd86451396ef176bb6f9fb69
I20260928 03:10:02.053468 28560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:10:02.063338 28560 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:10:02.065047 28560 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:10:02.079121 28560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a192759ae7564e6da73d469c7ff11cab: Generated new TSK 0
I20260928 03:10:02.080101 28560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:10:02.096252 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 28477
W20260928 03:10:02.097859 28539 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.198:52391
W20260928 03:10:02.098799 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 28477: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:10:02.098956 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 28477
W20260928 03:10:02.117251 28462 connection.cc:535] client connection to 127.0.0.1:40895 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40895): Connection reset by peer (error 104)
2026-09-28T03:10:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2039 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:10:02 dist-test-slave-fjww krb5kdc[28575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:10:02 dist-test-slave-fjww krb5kdc[28575](info): set up 2 sockets
Sep 28 03:10:02 dist-test-slave-fjww krb5kdc[28575](info): commencing operation
krb5kdc: starting...
W20260928 03:10:03.316727 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:10:03 dist-test-slave-fjww krb5kdc[28575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:10:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:10:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 03:10:03.596338 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34575
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:10:03.944079 28588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:10:03.944470 28588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:10:03.944649 28588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:10:03.952473 28588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:10:03.952656 28588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:10:03.952775 28588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:10:03.952885 28588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:10:03.952965 28588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:10:03.963636 28588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34575
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:10:03.965062 28588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:10:03.966495 28588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:10:03.979696 28594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:03.979795 28593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:03.979828 28596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:10:03.981513 28588 server_base.cc:1034] running on GCE node
I20260928 03:10:03.982554 28588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:10:03.984382 28588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:10:03.985636 28588 hybrid_clock.cc:648] HybridClock initialized: now 1790565003985534 us; error 102 us; skew 500 ppm
I20260928 03:10:03.986079 28588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:10:03 dist-test-slave-fjww krb5kdc[28575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:10:03.989996 28588 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 03:10:03.991854 28588 webserver.cc:460] Webserver started at http://127.0.0.1:46295/ using document root <none> and password file <none>
I20260928 03:10:03.992635 28588 fs_manager.cc:359] Metadata directory not provided
I20260928 03:10:03.992748 28588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:10:03.993221 28588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:10:03.996856 28588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "5bb766461983407f9772b951c07e0ac3"
format_stamp: "Formatted at 2026-09-28 03:10:03 on dist-test-slave-fjww"
server_key: "d7159aba443d68998b22411e4532d09b"
server_key_iv: "6e639b1c23ec1f137ead8a7e37f293a8"
server_key_version: "encryptionkey@0"
I20260928 03:10:03.997741 28588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "5bb766461983407f9772b951c07e0ac3"
format_stamp: "Formatted at 2026-09-28 03:10:03 on dist-test-slave-fjww"
server_key: "d7159aba443d68998b22411e4532d09b"
server_key_iv: "6e639b1c23ec1f137ead8a7e37f293a8"
server_key_version: "encryptionkey@0"
I20260928 03:10:04.004000 28588 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 03:10:04.008136 28603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:10:04.010066 28588 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:10:04.010346 28588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "5bb766461983407f9772b951c07e0ac3"
format_stamp: "Formatted at 2026-09-28 03:10:03 on dist-test-slave-fjww"
server_key: "d7159aba443d68998b22411e4532d09b"
server_key_iv: "6e639b1c23ec1f137ead8a7e37f293a8"
server_key_version: "encryptionkey@0"
I20260928 03:10:04.010612 28588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:10:04.017892 28588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:10:04.023291 28588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:10:04.023728 28588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:10:04.044198 28588 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34575
I20260928 03:10:04.044219 28654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34575 every 8 connection(s)
I20260928 03:10:04.045717 28588 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:10:04.049914 28655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:10:04.054530 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 28588
I20260928 03:10:04.054689 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:10:04.055217 6912 external_mini_cluster.cc:1428] Setting key fd3fb0906e1742b3a1086b346f18fab1
I20260928 03:10:04.064110 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:04.057347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:37217 (local address 127.0.0.1:34575)
0928 03:10:04.058209 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:04.058252 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:04.058297 (+ 45us) server_negotiation.cc:407] Connection header received
0928 03:10:04.060099 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:04.060125 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:04.060569 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:04.061045 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:04.062180 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:04.062186 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:04.062200 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:10:04.062216 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:04.062350 (+ 134us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:10:04.062398 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:04.062675 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:04.062818 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 03:10:04.063027 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":388,"threads_started":1}
I20260928 03:10:04.064254 28655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3: Bootstrap starting.
I20260928 03:10:04.068334 28655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3: Neither blocks nor log segments found. Creating new log.
I20260928 03:10:04.068706 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:10:04.069535 28655 log.cc:824] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3: Log is configured to *not* fsync() on all Append() calls
I20260928 03:10:04.072508 28655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3: No bootstrap required, opened a new log
I20260928 03:10:04.077009 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:04.073892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:52749 (local address 127.0.0.1:34575)
0928 03:10:04.074156 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:04.074167 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:04.074635 (+ 468us) server_negotiation.cc:407] Connection header received
0928 03:10:04.074952 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:04.074956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:04.075016 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:04.075111 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:04.076228 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:04.076232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:04.076235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:10:04.076252 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:04.076299 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:10:04.076316 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:04.076427 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:04.076567 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 03:10:04.076618 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 03:10:04.081579 28655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb766461983407f9772b951c07e0ac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34575 } }
I20260928 03:10:04.082028 28655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:10:04.082145 28655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bb766461983407f9772b951c07e0ac3, State: Initialized, Role: FOLLOWER
I20260928 03:10:04.082772 28655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb766461983407f9772b951c07e0ac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34575 } }
I20260928 03:10:04.083058 28655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:10:04.083212 28655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:10:04.083396 28655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:10:04.085816 28655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb766461983407f9772b951c07e0ac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34575 } }
I20260928 03:10:04.086393 28655 leader_election.cc:302] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bb766461983407f9772b951c07e0ac3; no voters:
I20260928 03:10:04.087126 28655 leader_election.cc:288] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:10:04.087311 28665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:10:04.088356 28665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3 [term 1 LEADER]: Becoming Leader. State: Replica: 5bb766461983407f9772b951c07e0ac3, State: Running, Role: LEADER
I20260928 03:10:04.088965 28665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb766461983407f9772b951c07e0ac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34575 } }
I20260928 03:10:04.089843 28655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:10:04.092273 28667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bb766461983407f9772b951c07e0ac3. Latest consensus state: current_term: 1 leader_uuid: "5bb766461983407f9772b951c07e0ac3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb766461983407f9772b951c07e0ac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34575 } } }
I20260928 03:10:04.092777 28667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3 [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:04.093303 28666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bb766461983407f9772b951c07e0ac3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb766461983407f9772b951c07e0ac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34575 } } }
I20260928 03:10:04.093636 28666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3 [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:04.095167 28674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:10:04.101971 28674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:10:04.112244 28674 catalog_manager.cc:1347] Generated new cluster ID: 4d828c0724814ba5a0517c9b26f01fd8
I20260928 03:10:04.112386 28674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:10:04.148453 28674 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:10:04.148972 28674 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:10:04.165232 28674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bb766461983407f9772b951c07e0ac3: Generated new TSK 0
I20260928 03:10:04.166433 28674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:10:04.171046 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 28588
W20260928 03:10:04.172739 28650 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.198:37217
W20260928 03:10:04.173645 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 28588: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:10:04.173826 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 28588
2026-09-28T03:10:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2070 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:10:04 dist-test-slave-fjww krb5kdc[28686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:10:04 dist-test-slave-fjww krb5kdc[28686](info): set up 2 sockets
Sep 28 03:10:04 dist-test-slave-fjww krb5kdc[28686](info): commencing operation
krb5kdc: starting...
W20260928 03:10:05.344157 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.006s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:10:05 dist-test-slave-fjww krb5kdc[28686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:10:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:10:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:10:05.627566 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:40655
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:40655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:10:05.990172 28699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:10:05.990496 28699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:10:05.990666 28699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:10:05.998451 28699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:10:05.998584 28699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:10:05.998672 28699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:10:05.998786 28699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:10:05.998862 28699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:10:06.010129 28699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:40655
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:40655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:10:06.011742 28699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:10:06.013188 28699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:10:06.026239 28707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:06.026196 28704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:06.026288 28705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:10:06.027513 28699 server_base.cc:1034] running on GCE node
I20260928 03:10:06.028679 28699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:10:06.031049 28699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:10:06.032279 28699 hybrid_clock.cc:648] HybridClock initialized: now 1790565006032118 us; error 134 us; skew 500 ppm
I20260928 03:10:06.032764 28699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:10:06 dist-test-slave-fjww krb5kdc[28686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565006, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:10:06.037385 28699 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:10:06.039137 28699 webserver.cc:460] Webserver started at http://127.6.192.62:37831/ using document root <none> and password file <none>
I20260928 03:10:06.039976 28699 fs_manager.cc:359] Metadata directory not provided
I20260928 03:10:06.040134 28699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:10:06.040468 28699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:10:06.043711 28699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "fa18ae4f94d248b9a01faef98b431472"
format_stamp: "Formatted at 2026-09-28 03:10:06 on dist-test-slave-fjww"
server_key: "d622131003fca1dee6af964d1ed5cc8f"
server_key_iv: "664a6733356591270316f5f5d031d4b9"
server_key_version: "encryptionkey@0"
I20260928 03:10:06.044553 28699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "fa18ae4f94d248b9a01faef98b431472"
format_stamp: "Formatted at 2026-09-28 03:10:06 on dist-test-slave-fjww"
server_key: "d622131003fca1dee6af964d1ed5cc8f"
server_key_iv: "664a6733356591270316f5f5d031d4b9"
server_key_version: "encryptionkey@0"
I20260928 03:10:06.050408 28699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 03:10:06.054872 28714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:10:06.056561 28699 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:10:06.056777 28699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "fa18ae4f94d248b9a01faef98b431472"
format_stamp: "Formatted at 2026-09-28 03:10:06 on dist-test-slave-fjww"
server_key: "d622131003fca1dee6af964d1ed5cc8f"
server_key_iv: "664a6733356591270316f5f5d031d4b9"
server_key_version: "encryptionkey@0"
I20260928 03:10:06.056995 28699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:10:06.073068 28699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:10:06.078135 28699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:10:06.078473 28699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:10:06.099084 28699 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:40655
I20260928 03:10:06.099107 28765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:40655 every 8 connection(s)
I20260928 03:10:06.100656 28699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:10:06.104789 28766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:10:06.108377 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 28699
I20260928 03:10:06.108574 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:10:06.109140 6912 external_mini_cluster.cc:1428] Setting key fc08393a29d68bf4cc85bc6734ffe6a5
I20260928 03:10:06.117856 28766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472: Bootstrap starting.
I20260928 03:10:06.121373 28766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472: Neither blocks nor log segments found. Creating new log.
I20260928 03:10:06.122689 28766 log.cc:824] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472: Log is configured to *not* fsync() on all Append() calls
Sep 28 03:10:06 dist-test-slave-fjww krb5kdc[28686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:10:06.125990 28766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472: No bootstrap required, opened a new log
I20260928 03:10:06.134794 28766 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa18ae4f94d248b9a01faef98b431472" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40655 } }
I20260928 03:10:06.135243 28766 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:10:06.135380 28766 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa18ae4f94d248b9a01faef98b431472, State: Initialized, Role: FOLLOWER
I20260928 03:10:06.135651 28769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:06.111353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45131 (local address 127.6.192.62:40655)
0928 03:10:06.112246 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:06.112261 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:06.112309 (+ 48us) server_negotiation.cc:407] Connection header received
0928 03:10:06.114217 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:06.114243 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:06.114737 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:06.115200 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:06.116976 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:06.118574 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:06.120165 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:06.121064 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:06.121332 (+ 268us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:06.126455 (+ 5123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:06.126467 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:06.126489 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:06.126541 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:06.130301 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:06.131338 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:06.131346 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:06.131354 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:06.131654 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:06.132219 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:06.132223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:06.132225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:06.132899 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:06.133303 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:06.133917 (+ 614us) server_negotiation.cc:299] Negotiation successful
0928 03:10:06.134332 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":370,"threads_started":1}
I20260928 03:10:06.136126 28766 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa18ae4f94d248b9a01faef98b431472" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40655 } }
I20260928 03:10:06.136452 28766 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:10:06.136590 28766 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:10:06.136765 28766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:10:06.139863 28766 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa18ae4f94d248b9a01faef98b431472" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40655 } }
I20260928 03:10:06.140478 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:10:06.140446 28766 leader_election.cc:302] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa18ae4f94d248b9a01faef98b431472; no voters:
I20260928 03:10:06.141099 28766 leader_election.cc:288] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:10:06.141255 28771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:10:06.141979 28771 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472 [term 1 LEADER]: Becoming Leader. State: Replica: fa18ae4f94d248b9a01faef98b431472, State: Running, Role: LEADER
I20260928 03:10:06.142627 28771 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa18ae4f94d248b9a01faef98b431472" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40655 } }
I20260928 03:10:06.144701 28771 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa18ae4f94d248b9a01faef98b431472. Latest consensus state: current_term: 1 leader_uuid: "fa18ae4f94d248b9a01faef98b431472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa18ae4f94d248b9a01faef98b431472" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40655 } } }
I20260928 03:10:06.145022 28771 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472 [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:06.145658 28773 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa18ae4f94d248b9a01faef98b431472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa18ae4f94d248b9a01faef98b431472" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 40655 } } }
I20260928 03:10:06.146096 28773 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472 [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:06.157739 28766 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:10:06.160462 28776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:10:06.172276 28776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:10:06.196748 28776 catalog_manager.cc:1347] Generated new cluster ID: 0733d036d47f48e98cc12802c55343b3
I20260928 03:10:06.196929 28776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:10:06.202940 28769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:06.169384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49245 (local address 127.6.192.62:40655)
0928 03:10:06.169657 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:06.169668 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:06.177018 (+ 7350us) server_negotiation.cc:407] Connection header received
0928 03:10:06.178295 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:06.178306 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:06.178445 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:06.178560 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:06.181727 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:06.183507 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:06.186429 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:06.187532 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:06.187682 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:06.192805 (+ 5123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:06.192817 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:06.192826 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:06.192905 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:06.198712 (+ 5807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:06.200006 (+ 1294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:06.200013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:06.200017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:06.200178 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:06.200817 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:06.200823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:06.200826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:06.201285 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:06.201532 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:06.202330 (+ 798us) server_negotiation.cc:299] Negotiation successful
0928 03:10:06.202546 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 03:10:06.219764 28776 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:10:06.221825 28776 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:10:06.235373 28776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa18ae4f94d248b9a01faef98b431472: Generated new TSK 0
I20260928 03:10:06.236392 28776 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:10:06.244925 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 28699
I20260928 03:10:06.246105 28761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:10:06.339597 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 28699
W20260928 03:10:06.358520 28684 connection.cc:535] client connection to 127.6.192.62:40655 recv error: Network error: failed to read from TLS socket (remote: 127.6.192.62:40655): Connection reset by peer (error 104)
2026-09-28T03:10:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2171 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:10:06 dist-test-slave-fjww krb5kdc[28798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:10:06 dist-test-slave-fjww krb5kdc[28798](info): set up 2 sockets
Sep 28 03:10:06 dist-test-slave-fjww krb5kdc[28798](info): commencing operation
krb5kdc: starting...
W20260928 03:10:07.488564 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:10:07 dist-test-slave-fjww krb5kdc[28798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:10:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:10:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:10:07.772368 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:38537
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:38537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:10:08.134681 28811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:10:08.135032 28811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:10:08.135245 28811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:10:08.143821 28811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:10:08.143954 28811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:10:08.144061 28811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:10:08.144205 28811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:10:08.144285 28811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:10:08.155645 28811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:38537
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:38537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:10:08.157052 28811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:10:08.158524 28811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:10:08.171568 28819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:08.171591 28816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:08.171615 28817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:10:08.173044 28811 server_base.cc:1034] running on GCE node
I20260928 03:10:08.173921 28811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:10:08.175550 28811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:10:08.176801 28811 hybrid_clock.cc:648] HybridClock initialized: now 1790565008176756 us; error 38 us; skew 500 ppm
I20260928 03:10:08.177178 28811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:10:08 dist-test-slave-fjww krb5kdc[28798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565008, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:10:08.180665 28811 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:10:08.182368 28811 webserver.cc:460] Webserver started at http://127.6.192.62:32833/ using document root <none> and password file <none>
I20260928 03:10:08.183116 28811 fs_manager.cc:359] Metadata directory not provided
I20260928 03:10:08.183245 28811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:10:08.183630 28811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:10:08.186988 28811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "5f92b41536bc4a43ad3f34af79e45ff8"
format_stamp: "Formatted at 2026-09-28 03:10:08 on dist-test-slave-fjww"
server_key: "89d118b4a8f743f8092be463d0dcf64f"
server_key_iv: "0d259cade2e38b64908afbf516d1c3a4"
server_key_version: "encryptionkey@0"
I20260928 03:10:08.187881 28811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "5f92b41536bc4a43ad3f34af79e45ff8"
format_stamp: "Formatted at 2026-09-28 03:10:08 on dist-test-slave-fjww"
server_key: "89d118b4a8f743f8092be463d0dcf64f"
server_key_iv: "0d259cade2e38b64908afbf516d1c3a4"
server_key_version: "encryptionkey@0"
I20260928 03:10:08.194260 28811 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260928 03:10:08.198038 28826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:10:08.199661 28811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 03:10:08.199857 28811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "5f92b41536bc4a43ad3f34af79e45ff8"
format_stamp: "Formatted at 2026-09-28 03:10:08 on dist-test-slave-fjww"
server_key: "89d118b4a8f743f8092be463d0dcf64f"
server_key_iv: "0d259cade2e38b64908afbf516d1c3a4"
server_key_version: "encryptionkey@0"
I20260928 03:10:08.200071 28811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:10:08.214923 28811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:10:08.219926 28811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:10:08.220263 28811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:10:08.240208 28811 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:38537
I20260928 03:10:08.240257 28877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:38537 every 8 connection(s)
I20260928 03:10:08.241684 28811 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:10:08.245668 28878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:10:08.251814 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 28811
I20260928 03:10:08.252025 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:10:08.252568 6912 external_mini_cluster.cc:1428] Setting key a3fb329e82dd69d22301ce49faf6dc65
I20260928 03:10:08.257509 28878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8: Bootstrap starting.
I20260928 03:10:08.261557 28878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8: Neither blocks nor log segments found. Creating new log.
I20260928 03:10:08.262836 28878 log.cc:824] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8: Log is configured to *not* fsync() on all Append() calls
I20260928 03:10:08.266543 28878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8: No bootstrap required, opened a new log
I20260928 03:10:08.267719 28881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:08.254680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48317 (local address 127.6.192.62:38537)
0928 03:10:08.255669 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:08.255682 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:08.255725 (+ 43us) server_negotiation.cc:407] Connection header received
0928 03:10:08.257477 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:08.257503 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:08.258037 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:08.258387 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:08.259028 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:08.260537 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:08.262510 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:08.263283 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:08.263597 (+ 314us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:08.264849 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:08.264859 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:08.264882 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:10:08.265604 (+ 722us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:08.265711 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:10:08.265743 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:08.265905 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:08.266199 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 03:10:08.266374 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":457,"threads_started":1}
I20260928 03:10:08.272044 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:10:08.275696 28878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f92b41536bc4a43ad3f34af79e45ff8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38537 } }
I20260928 03:10:08.276115 28878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:10:08.276237 28878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f92b41536bc4a43ad3f34af79e45ff8, State: Initialized, Role: FOLLOWER
I20260928 03:10:08.277052 28878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f92b41536bc4a43ad3f34af79e45ff8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38537 } }
I20260928 03:10:08.277347 28878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:10:08.277526 28878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:10:08.277734 28878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:10:08.280997 28878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f92b41536bc4a43ad3f34af79e45ff8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38537 } }
I20260928 03:10:08.281661 28878 leader_election.cc:302] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f92b41536bc4a43ad3f34af79e45ff8; no voters:
I20260928 03:10:08.282410 28878 leader_election.cc:288] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:10:08.282521 28888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:10:08.283270 28888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8 [term 1 LEADER]: Becoming Leader. State: Replica: 5f92b41536bc4a43ad3f34af79e45ff8, State: Running, Role: LEADER
I20260928 03:10:08.284008 28888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f92b41536bc4a43ad3f34af79e45ff8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38537 } }
I20260928 03:10:08.284462 28881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:08.276687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33615 (local address 127.6.192.62:38537)
0928 03:10:08.276900 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:08.276908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:08.277295 (+ 387us) server_negotiation.cc:407] Connection header received
0928 03:10:08.277537 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:08.277542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:08.277607 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:08.277733 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:08.278428 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:08.279351 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:08.281297 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:08.281802 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:08.282067 (+ 265us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:08.283554 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:08.283559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:08.283564 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:10:08.283585 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:08.283659 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:10:08.283681 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:08.283780 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:08.284051 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 03:10:08.284105 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 03:10:08.284978 28878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:10:08.287628 28889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f92b41536bc4a43ad3f34af79e45ff8. Latest consensus state: current_term: 1 leader_uuid: "5f92b41536bc4a43ad3f34af79e45ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f92b41536bc4a43ad3f34af79e45ff8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38537 } } }
I20260928 03:10:08.288133 28889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8 [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:08.293932 28895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:10:08.292368 28890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f92b41536bc4a43ad3f34af79e45ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f92b41536bc4a43ad3f34af79e45ff8" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38537 } } }
I20260928 03:10:08.296955 28890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8 [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:08.301546 28895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:10:08.315518 28895 catalog_manager.cc:1347] Generated new cluster ID: 37f54054302a49b19e69cbf4c77c9ec2
I20260928 03:10:08.315702 28895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:10:08.328367 28895 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:10:08.330331 28895 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:10:08.341933 28895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f92b41536bc4a43ad3f34af79e45ff8: Generated new TSK 0
I20260928 03:10:08.342849 28895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:10:08.358536 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 28811
W20260928 03:10:08.359848 28873 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48317
W20260928 03:10:08.360692 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 28811: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:10:08.360853 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 28811
W20260928 03:10:08.375245 28796 connection.cc:535] client connection to 127.6.192.62:38537 recv error: Network error: failed to read from TLS socket (remote: 127.6.192.62:38537): Connection reset by peer (error 104)
2026-09-28T03:10:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2015 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:10:08 dist-test-slave-fjww krb5kdc[28909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:10:08 dist-test-slave-fjww krb5kdc[28909](info): set up 2 sockets
Sep 28 03:10:08 dist-test-slave-fjww krb5kdc[28909](info): commencing operation
krb5kdc: starting...
W20260928 03:10:09.488168 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:10:09 dist-test-slave-fjww krb5kdc[28909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:10:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:10:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:10:09.775619 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:45757
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:38095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:45757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:10:10.129354 28922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:10:10.129757 28922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:10:10.129967 28922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:10:10.138409 28922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:10:10.138587 28922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:10:10.138703 28922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:10:10.138837 28922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:10:10.138924 28922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:10:10.149175 28922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:45757
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:45757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:10:10.150575 28922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:10:10.151999 28922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:10:10.164312 28930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:10.164359 28928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:10.164269 28927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:10:10.165791 28922 server_base.cc:1034] running on GCE node
I20260928 03:10:10.166746 28922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:10:10.168516 28922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:10:10.169818 28922 hybrid_clock.cc:648] HybridClock initialized: now 1790565010169747 us; error 57 us; skew 500 ppm
I20260928 03:10:10.170245 28922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:10:10 dist-test-slave-fjww krb5kdc[28909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565010, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:10:10.178773 28922 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:10:10.180490 28922 webserver.cc:460] Webserver started at http://127.6.192.62:39367/ using document root <none> and password file <none>
I20260928 03:10:10.181177 28922 fs_manager.cc:359] Metadata directory not provided
I20260928 03:10:10.181291 28922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:10:10.181553 28922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:10:10.184641 28922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "e379a80f30a642fc81a601f823ff33cd"
format_stamp: "Formatted at 2026-09-28 03:10:10 on dist-test-slave-fjww"
server_key: "6315df62732209adc83ab70234a8b570"
server_key_iv: "4db0c9aea18c4e7841857d3e0cbbd7b8"
server_key_version: "encryptionkey@0"
I20260928 03:10:10.185401 28922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "e379a80f30a642fc81a601f823ff33cd"
format_stamp: "Formatted at 2026-09-28 03:10:10 on dist-test-slave-fjww"
server_key: "6315df62732209adc83ab70234a8b570"
server_key_iv: "4db0c9aea18c4e7841857d3e0cbbd7b8"
server_key_version: "encryptionkey@0"
I20260928 03:10:10.190793 28922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 03:10:10.194561 28937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:10:10.196087 28922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 03:10:10.196236 28922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "e379a80f30a642fc81a601f823ff33cd"
format_stamp: "Formatted at 2026-09-28 03:10:10 on dist-test-slave-fjww"
server_key: "6315df62732209adc83ab70234a8b570"
server_key_iv: "4db0c9aea18c4e7841857d3e0cbbd7b8"
server_key_version: "encryptionkey@0"
I20260928 03:10:10.196421 28922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:10:10.202307 28922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:10:10.206833 28922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:10:10.207093 28922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:10:10.227154 28922 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:45757
I20260928 03:10:10.227197 28988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:45757 every 8 connection(s)
I20260928 03:10:10.228704 28922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:10:10.232966 28989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:10:10.234869 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 28922
I20260928 03:10:10.235055 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:10:10.235661 6912 external_mini_cluster.cc:1428] Setting key 493ff54859082387e2109d281e829f5a
I20260928 03:10:10.244791 28992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:10.237879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51597 (local address 127.6.192.62:45757)
0928 03:10:10.238916 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:10.238929 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:10.238984 (+ 55us) server_negotiation.cc:407] Connection header received
0928 03:10:10.240972 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:10.240999 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:10.241404 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:10.241791 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:10.242983 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:10.242990 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:10.243004 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:10:10.243024 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:10.243094 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:10:10.243119 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:10.243340 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:10.243521 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 03:10:10.243795 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":477,"threads_started":1}
I20260928 03:10:10.245946 28989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd: Bootstrap starting.
I20260928 03:10:10.248795 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:10:10.249472 28989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd: Neither blocks nor log segments found. Creating new log.
I20260928 03:10:10.250741 28989 log.cc:824] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd: Log is configured to *not* fsync() on all Append() calls
I20260928 03:10:10.253940 28989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd: No bootstrap required, opened a new log
I20260928 03:10:10.257071 28992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:10.254127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54445 (local address 127.6.192.62:45757)
0928 03:10:10.254362 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:10.254372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:10.254861 (+ 489us) server_negotiation.cc:407] Connection header received
0928 03:10:10.255100 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:10.255104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:10.255153 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:10.255246 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:10.256324 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:10.256328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:10.256334 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:10:10.256350 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:10.256395 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:10:10.256412 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:10.256509 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:10.256689 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 03:10:10.256739 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 03:10:10.263866 28989 raft_consensus.cc:348] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e379a80f30a642fc81a601f823ff33cd" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 45757 } }
I20260928 03:10:10.264339 28989 raft_consensus.cc:374] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:10:10.264456 28989 raft_consensus.cc:729] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e379a80f30a642fc81a601f823ff33cd, State: Initialized, Role: FOLLOWER
I20260928 03:10:10.265151 28989 consensus_queue.cc:260] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e379a80f30a642fc81a601f823ff33cd" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 45757 } }
I20260928 03:10:10.265419 28989 raft_consensus.cc:388] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:10:10.265573 28989 raft_consensus.cc:482] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:10:10.265767 28989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:10:10.268385 28989 raft_consensus.cc:504] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e379a80f30a642fc81a601f823ff33cd" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 45757 } }
I20260928 03:10:10.268944 28989 leader_election.cc:302] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e379a80f30a642fc81a601f823ff33cd; no voters:
I20260928 03:10:10.269603 28989 leader_election.cc:288] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:10:10.269879 28999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:10:10.270695 28999 raft_consensus.cc:686] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd [term 1 LEADER]: Becoming Leader. State: Replica: e379a80f30a642fc81a601f823ff33cd, State: Running, Role: LEADER
I20260928 03:10:10.271267 28999 consensus_queue.cc:237] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e379a80f30a642fc81a601f823ff33cd" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 45757 } }
I20260928 03:10:10.271862 28989 sys_catalog.cc:564] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:10:10.274469 29000 sys_catalog.cc:455] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e379a80f30a642fc81a601f823ff33cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e379a80f30a642fc81a601f823ff33cd" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 45757 } } }
I20260928 03:10:10.274955 29000 sys_catalog.cc:458] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:10.275380 29001 sys_catalog.cc:455] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e379a80f30a642fc81a601f823ff33cd. Latest consensus state: current_term: 1 leader_uuid: "e379a80f30a642fc81a601f823ff33cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e379a80f30a642fc81a601f823ff33cd" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 45757 } } }
I20260928 03:10:10.275722 29001 sys_catalog.cc:458] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:10.280900 29005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:10:10.285521 29005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:10:10.295014 29005 catalog_manager.cc:1347] Generated new cluster ID: d97a12d937cd4ed4b4fe14c9202ef466
I20260928 03:10:10.295151 29005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:10:10.307897 29005 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:10:10.308379 29005 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:10:10.317767 29005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e379a80f30a642fc81a601f823ff33cd: Generated new TSK 0
I20260928 03:10:10.318956 29005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:10:10.329852 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 28922
W20260928 03:10:10.331753 28984 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51597
W20260928 03:10:10.332628 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 28922: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:10:10.332983 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 28922
2026-09-28T03:10:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1975 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:10:10 dist-test-slave-fjww krb5kdc[29020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:10:10 dist-test-slave-fjww krb5kdc[29020](info): set up 2 sockets
Sep 28 03:10:10 dist-test-slave-fjww krb5kdc[29020](info): commencing operation
krb5kdc: starting...
W20260928 03:10:11.568755 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:10:11 dist-test-slave-fjww krb5kdc[29020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:10:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:10:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:10:11.848121 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:36597
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:46241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:36597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:10:12.195830 29033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:10:12.196130 29033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:10:12.196302 29033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:10:12.203719 29033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:10:12.203848 29033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:10:12.203943 29033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:10:12.204030 29033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:10:12.204113 29033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:10:12.214500 29033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:46241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:36597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:36597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:10:12.216020 29033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:10:12.217413 29033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:10:12.229756 29038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:12.229681 29041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:12.229786 29039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:10:12.230770 29033 server_base.cc:1034] running on GCE node
I20260928 03:10:12.231689 29033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:10:12.233265 29033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:10:12.234503 29033 hybrid_clock.cc:648] HybridClock initialized: now 1790565012234455 us; error 48 us; skew 500 ppm
I20260928 03:10:12.234871 29033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:10:12 dist-test-slave-fjww krb5kdc[29020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565012, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:10:12.238310 29033 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:10:12.240185 29033 webserver.cc:460] Webserver started at http://127.6.192.62:38263/ using document root <none> and password file <none>
I20260928 03:10:12.240855 29033 fs_manager.cc:359] Metadata directory not provided
I20260928 03:10:12.240971 29033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:10:12.241228 29033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:10:12.244415 29033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "07afef56f1ec4fe99cbdf619806d7584"
format_stamp: "Formatted at 2026-09-28 03:10:12 on dist-test-slave-fjww"
server_key: "6eea1e95816dabe56d854001728aa166"
server_key_iv: "3aaa1a4ad62961f005a95422c25c86ed"
server_key_version: "encryptionkey@0"
I20260928 03:10:12.245206 29033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "07afef56f1ec4fe99cbdf619806d7584"
format_stamp: "Formatted at 2026-09-28 03:10:12 on dist-test-slave-fjww"
server_key: "6eea1e95816dabe56d854001728aa166"
server_key_iv: "3aaa1a4ad62961f005a95422c25c86ed"
server_key_version: "encryptionkey@0"
I20260928 03:10:12.250761 29033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 03:10:12.254580 29048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:10:12.256253 29033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 03:10:12.256508 29033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "07afef56f1ec4fe99cbdf619806d7584"
format_stamp: "Formatted at 2026-09-28 03:10:12 on dist-test-slave-fjww"
server_key: "6eea1e95816dabe56d854001728aa166"
server_key_iv: "3aaa1a4ad62961f005a95422c25c86ed"
server_key_version: "encryptionkey@0"
I20260928 03:10:12.256697 29033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:10:12.287281 29033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:10:12.292104 29033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:10:12.292424 29033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:10:12.313374 29033 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:36597
I20260928 03:10:12.313412 29099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:36597 every 8 connection(s)
I20260928 03:10:12.314823 29033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:10:12.317442 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 29033
I20260928 03:10:12.317627 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:10:12.318200 6912 external_mini_cluster.cc:1428] Setting key 44c034bfab4781cf47af6a2b58a08b4c
I20260928 03:10:12.319584 29100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 03:10:12 dist-test-slave-fjww krb5kdc[29020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:10:12.334887 29100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584: Bootstrap starting.
I20260928 03:10:12.338573 29100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584: Neither blocks nor log segments found. Creating new log.
I20260928 03:10:12.340014 29100 log.cc:824] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584: Log is configured to *not* fsync() on all Append() calls
I20260928 03:10:12.343187 29103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:12.320108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37583 (local address 127.6.192.62:36597)
0928 03:10:12.320979 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:12.320992 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:12.321035 (+ 43us) server_negotiation.cc:407] Connection header received
0928 03:10:12.322613 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:12.322636 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:12.323067 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:12.323427 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:12.325294 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:12.326748 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:12.328577 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:12.329277 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:12.329457 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:12.334745 (+ 5288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:12.334754 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:12.334772 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:12.334816 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:12.338203 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:12.339200 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:12.339207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:12.339214 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:12.339427 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:12.340008 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:12.340013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:12.340015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:12.340863 (+ 848us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:12.341281 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:12.341645 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 03:10:12.342092 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":435,"threads_started":1}
I20260928 03:10:12.343773 29100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584: No bootstrap required, opened a new log
I20260928 03:10:12.347328 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:10:12.355827 29100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07afef56f1ec4fe99cbdf619806d7584" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36597 } }
I20260928 03:10:12.356472 29100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:10:12.356603 29100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07afef56f1ec4fe99cbdf619806d7584, State: Initialized, Role: FOLLOWER
I20260928 03:10:12.357419 29100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07afef56f1ec4fe99cbdf619806d7584" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36597 } }
I20260928 03:10:12.357731 29100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:10:12.357918 29100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:10:12.358121 29100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:10:12.361728 29100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07afef56f1ec4fe99cbdf619806d7584" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36597 } }
I20260928 03:10:12.362336 29100 leader_election.cc:302] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07afef56f1ec4fe99cbdf619806d7584; no voters:
I20260928 03:10:12.363102 29100 leader_election.cc:288] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:10:12.363225 29110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:10:12.366163 29100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:10:12.367753 29110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584 [term 1 LEADER]: Becoming Leader. State: Replica: 07afef56f1ec4fe99cbdf619806d7584, State: Running, Role: LEADER
I20260928 03:10:12.368762 29110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07afef56f1ec4fe99cbdf619806d7584" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36597 } }
I20260928 03:10:12.372855 29112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07afef56f1ec4fe99cbdf619806d7584. Latest consensus state: current_term: 1 leader_uuid: "07afef56f1ec4fe99cbdf619806d7584" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07afef56f1ec4fe99cbdf619806d7584" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36597 } } }
I20260928 03:10:12.373440 29112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584 [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:12.373342 29111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07afef56f1ec4fe99cbdf619806d7584" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07afef56f1ec4fe99cbdf619806d7584" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 36597 } } }
I20260928 03:10:12.373674 29111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584 [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:12.372854 29103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:12.353806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50659 (local address 127.6.192.62:36597)
0928 03:10:12.354059 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:12.354066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:12.354695 (+ 629us) server_negotiation.cc:407] Connection header received
0928 03:10:12.354903 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:12.354907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:12.354977 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:12.355064 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:12.357286 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:12.358405 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:12.360459 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:12.361038 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:12.361245 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:12.364675 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:12.364682 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:12.364690 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:12.364743 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:12.368331 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:12.369390 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:12.369397 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:12.369404 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:12.369585 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:12.370118 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:12.370129 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:12.370133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:12.370648 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:12.370973 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:12.372073 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0928 03:10:12.372354 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 03:10:12.376971 29118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:10:12.385079 29118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:10:12.394695 29118 catalog_manager.cc:1347] Generated new cluster ID: eae36ecb38024f2186f1c3479a2a94b6
I20260928 03:10:12.394824 29118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:10:12.419207 29118 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:10:12.421306 29118 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:10:12.433197 29118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07afef56f1ec4fe99cbdf619806d7584: Generated new TSK 0
I20260928 03:10:12.434151 29118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:10:12.443377 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 29033
I20260928 03:10:12.444620 29095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:10:12.544082 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 29033
2026-09-28T03:10:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:10:12 dist-test-slave-fjww krb5kdc[29132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:10:12 dist-test-slave-fjww krb5kdc[29132](info): set up 2 sockets
Sep 28 03:10:12 dist-test-slave-fjww krb5kdc[29132](info): commencing operation
krb5kdc: starting...
W20260928 03:10:13.676273 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:10:13 dist-test-slave-fjww krb5kdc[29132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:10:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:10:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:10:13.957293 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:42243
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:38171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:42243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:10:14.303189 29145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:10:14.303612 29145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:10:14.303776 29145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:10:14.311287 29145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:10:14.311510 29145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:10:14.311630 29145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:10:14.311743 29145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:10:14.311818 29145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:10:14.322254 29145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:42243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:42243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:10:14.323772 29145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:10:14.325176 29145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:10:14.338981 29150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:14.339040 29151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:14.338953 29153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:10:14.340015 29145 server_base.cc:1034] running on GCE node
I20260928 03:10:14.340809 29145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:10:14.342618 29145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:10:14.343854 29145 hybrid_clock.cc:648] HybridClock initialized: now 1790565014343786 us; error 67 us; skew 500 ppm
I20260928 03:10:14.344252 29145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:10:14 dist-test-slave-fjww krb5kdc[29132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565014, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:10:14.347798 29145 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:10:14.349640 29145 webserver.cc:460] Webserver started at http://127.6.192.62:32957/ using document root <none> and password file <none>
I20260928 03:10:14.350363 29145 fs_manager.cc:359] Metadata directory not provided
I20260928 03:10:14.350505 29145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:10:14.350828 29145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:10:14.354079 29145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "63e70c07a17341b4a4de2e50c5d5f3f0"
format_stamp: "Formatted at 2026-09-28 03:10:14 on dist-test-slave-fjww"
server_key: "28fbb7157b2c01b9c773f6a018683a9c"
server_key_iv: "b684e0b6fe8467cd4d2181217880b847"
server_key_version: "encryptionkey@0"
I20260928 03:10:14.354947 29145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "63e70c07a17341b4a4de2e50c5d5f3f0"
format_stamp: "Formatted at 2026-09-28 03:10:14 on dist-test-slave-fjww"
server_key: "28fbb7157b2c01b9c773f6a018683a9c"
server_key_iv: "b684e0b6fe8467cd4d2181217880b847"
server_key_version: "encryptionkey@0"
I20260928 03:10:14.360888 29145 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 03:10:14.364980 29160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:10:14.366636 29145 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 03:10:14.366891 29145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "63e70c07a17341b4a4de2e50c5d5f3f0"
format_stamp: "Formatted at 2026-09-28 03:10:14 on dist-test-slave-fjww"
server_key: "28fbb7157b2c01b9c773f6a018683a9c"
server_key_iv: "b684e0b6fe8467cd4d2181217880b847"
server_key_version: "encryptionkey@0"
I20260928 03:10:14.367151 29145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:10:14.403349 29145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:10:14.407953 29145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:10:14.408288 29145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:10:14.428556 29145 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:42243
I20260928 03:10:14.428597 29211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:42243 every 8 connection(s)
I20260928 03:10:14.430068 29145 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:10:14.434064 29212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:10:14.436251 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 29145
I20260928 03:10:14.436443 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:10:14.436980 6912 external_mini_cluster.cc:1428] Setting key 02d19d3f51062b93ed59dc8a324210b6
I20260928 03:10:14.447379 29212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0: Bootstrap starting.
I20260928 03:10:14.451328 29212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0: Neither blocks nor log segments found. Creating new log.
I20260928 03:10:14.451719 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:14.439052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51839 (local address 127.6.192.62:42243)
0928 03:10:14.439990 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:14.440006 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:14.440049 (+ 43us) server_negotiation.cc:407] Connection header received
0928 03:10:14.441863 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:14.441889 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:14.442321 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:14.442713 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:14.443499 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:14.444932 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:14.446852 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:14.447683 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:14.447824 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:14.449142 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:14.449152 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:14.449168 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:10:14.449767 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:14.449847 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:10:14.449877 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:14.449975 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:14.450228 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 03:10:14.450529 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":448,"threads_started":1}
I20260928 03:10:14.452632 29212 log.cc:824] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0: Log is configured to *not* fsync() on all Append() calls
I20260928 03:10:14.456010 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:10:14.456076 29212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0: No bootstrap required, opened a new log
I20260928 03:10:14.466944 29212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e70c07a17341b4a4de2e50c5d5f3f0" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 42243 } }
I20260928 03:10:14.467566 29212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:10:14.467692 29212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e70c07a17341b4a4de2e50c5d5f3f0, State: Initialized, Role: FOLLOWER
I20260928 03:10:14.468516 29212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e70c07a17341b4a4de2e50c5d5f3f0" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 42243 } }
I20260928 03:10:14.468743 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:14.461545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45355 (local address 127.6.192.62:42243)
0928 03:10:14.461784 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:14.461790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:14.462254 (+ 464us) server_negotiation.cc:407] Connection header received
0928 03:10:14.462421 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:14.462427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:14.462525 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:14.462648 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:14.463186 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:14.464424 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:14.465848 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:14.466508 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:14.466663 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:14.467876 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:14.467883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:14.467887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:10:14.467912 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:14.467963 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:10:14.467985 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:14.468084 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:14.468272 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 03:10:14.468378 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 03:10:14.468885 29212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:10:14.469053 29212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:10:14.469251 29212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:10:14.471983 29212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e70c07a17341b4a4de2e50c5d5f3f0" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 42243 } }
I20260928 03:10:14.472553 29212 leader_election.cc:302] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63e70c07a17341b4a4de2e50c5d5f3f0; no voters:
I20260928 03:10:14.473161 29212 leader_election.cc:288] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:10:14.473301 29222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:10:14.474071 29222 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0 [term 1 LEADER]: Becoming Leader. State: Replica: 63e70c07a17341b4a4de2e50c5d5f3f0, State: Running, Role: LEADER
I20260928 03:10:14.474711 29222 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e70c07a17341b4a4de2e50c5d5f3f0" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 42243 } }
I20260928 03:10:14.475595 29212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:10:14.477566 29223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63e70c07a17341b4a4de2e50c5d5f3f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e70c07a17341b4a4de2e50c5d5f3f0" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 42243 } } }
I20260928 03:10:14.477931 29223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0 [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:14.478852 29227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:10:14.478701 29224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63e70c07a17341b4a4de2e50c5d5f3f0. Latest consensus state: current_term: 1 leader_uuid: "63e70c07a17341b4a4de2e50c5d5f3f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e70c07a17341b4a4de2e50c5d5f3f0" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 42243 } } }
I20260928 03:10:14.479081 29224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0 [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:14.488209 29227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:10:14.498396 29227 catalog_manager.cc:1347] Generated new cluster ID: 3b0227a41b6f414bac308a69d6e5a1b7
I20260928 03:10:14.498548 29227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:10:14.509562 29227 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:10:14.511564 29227 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:10:14.525504 29227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63e70c07a17341b4a4de2e50c5d5f3f0: Generated new TSK 0
I20260928 03:10:14.526782 29227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:10:14.538486 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 29145
W20260928 03:10:14.540021 29207 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51839
W20260928 03:10:14.540910 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 29145: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:10:14.541086 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 29145
2026-09-28T03:10:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1997 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:10:14 dist-test-slave-fjww krb5kdc[29243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:10:14 dist-test-slave-fjww krb5kdc[29243](info): set up 2 sockets
Sep 28 03:10:14 dist-test-slave-fjww krb5kdc[29243](info): commencing operation
krb5kdc: starting...
W20260928 03:10:15.784296 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:10:15 dist-test-slave-fjww krb5kdc[29243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:10:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:10:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:10:16.073290 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:37819
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:38693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:37819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:10:16.426481 29256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:10:16.426805 29256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:10:16.427006 29256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:10:16.434582 29256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:10:16.434716 29256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:10:16.434763 29256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:10:16.434849 29256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:10:16.434938 29256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:10:16.445447 29256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:38693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:37819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:37819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:10:16.446909 29256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:10:16.448361 29256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:10:16.461037 29264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:16.461105 29262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:16.461155 29261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:10:16.462765 29256 server_base.cc:1034] running on GCE node
I20260928 03:10:16.463887 29256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:10:16.465673 29256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:10:16.466931 29256 hybrid_clock.cc:648] HybridClock initialized: now 1790565016466848 us; error 72 us; skew 500 ppm
I20260928 03:10:16.467365 29256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:10:16 dist-test-slave-fjww krb5kdc[29243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565016, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:10:16.476001 29256 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:10:16.477947 29256 webserver.cc:460] Webserver started at http://127.6.192.62:45543/ using document root <none> and password file <none>
I20260928 03:10:16.478744 29256 fs_manager.cc:359] Metadata directory not provided
I20260928 03:10:16.478865 29256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:10:16.479197 29256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:10:16.482586 29256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "62e5dd81990f445fac0f60cb13b1dc40"
format_stamp: "Formatted at 2026-09-28 03:10:16 on dist-test-slave-fjww"
server_key: "3b857f7306f89575214d445c218557f2"
server_key_iv: "31f39762cc635466c1469446c8011a49"
server_key_version: "encryptionkey@0"
I20260928 03:10:16.483481 29256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "62e5dd81990f445fac0f60cb13b1dc40"
format_stamp: "Formatted at 2026-09-28 03:10:16 on dist-test-slave-fjww"
server_key: "3b857f7306f89575214d445c218557f2"
server_key_iv: "31f39762cc635466c1469446c8011a49"
server_key_version: "encryptionkey@0"
I20260928 03:10:16.489524 29256 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 03:10:16.493835 29271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:10:16.495831 29256 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 03:10:16.496062 29256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "62e5dd81990f445fac0f60cb13b1dc40"
format_stamp: "Formatted at 2026-09-28 03:10:16 on dist-test-slave-fjww"
server_key: "3b857f7306f89575214d445c218557f2"
server_key_iv: "31f39762cc635466c1469446c8011a49"
server_key_version: "encryptionkey@0"
I20260928 03:10:16.496325 29256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:10:16.503607 29256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:10:16.508317 29256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:10:16.508694 29256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:10:16.530184 29256 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:37819
I20260928 03:10:16.530227 29322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:37819 every 8 connection(s)
I20260928 03:10:16.532105 29256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:10:16.536834 29323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:10:16.541687 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 29256
I20260928 03:10:16.541884 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:10:16.542429 6912 external_mini_cluster.cc:1428] Setting key 11af55592cd2bf5f0b676e760baf7dd8
I20260928 03:10:16.549539 29323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40: Bootstrap starting.
I20260928 03:10:16.552414 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:16.544712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35877 (local address 127.6.192.62:37819)
0928 03:10:16.545757 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:16.545771 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:16.545828 (+ 57us) server_negotiation.cc:407] Connection header received
0928 03:10:16.548174 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:16.548219 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:16.548657 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:16.549160 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:16.550415 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:16.550423 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:16.550438 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:10:16.550456 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:16.550525 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:10:16.550555 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:16.550761 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:16.550887 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 03:10:16.551142 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":500,"threads_started":1}
I20260928 03:10:16.553579 29323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40: Neither blocks nor log segments found. Creating new log.
I20260928 03:10:16.554739 29323 log.cc:824] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40: Log is configured to *not* fsync() on all Append() calls
I20260928 03:10:16.556924 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:10:16.557948 29323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40: No bootstrap required, opened a new log
I20260928 03:10:16.563970 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:16.561389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40447 (local address 127.6.192.62:37819)
0928 03:10:16.561631 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:16.561637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:16.561924 (+ 287us) server_negotiation.cc:407] Connection header received
0928 03:10:16.562152 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:16.562156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:16.562208 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:16.562318 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:16.563262 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:16.563267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:16.563272 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 03:10:16.563292 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:16.563348 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 03:10:16.563367 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:16.563525 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:16.563618 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 03:10:16.563685 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 03:10:16.566962 29323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e5dd81990f445fac0f60cb13b1dc40" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 37819 } }
I20260928 03:10:16.567389 29323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 03:10:16.567584 29323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62e5dd81990f445fac0f60cb13b1dc40, State: Initialized, Role: FOLLOWER
I20260928 03:10:16.568228 29323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e5dd81990f445fac0f60cb13b1dc40" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 37819 } }
I20260928 03:10:16.568506 29323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 03:10:16.568658 29323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 03:10:16.568851 29323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:10:16.571512 29323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e5dd81990f445fac0f60cb13b1dc40" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 37819 } }
I20260928 03:10:16.572086 29323 leader_election.cc:302] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62e5dd81990f445fac0f60cb13b1dc40; no voters:
I20260928 03:10:16.572798 29323 leader_election.cc:288] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 03:10:16.573001 29333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:10:16.573803 29333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40 [term 1 LEADER]: Becoming Leader. State: Replica: 62e5dd81990f445fac0f60cb13b1dc40, State: Running, Role: LEADER
I20260928 03:10:16.574462 29333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e5dd81990f445fac0f60cb13b1dc40" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 37819 } }
I20260928 03:10:16.575139 29323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:10:16.577920 29334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62e5dd81990f445fac0f60cb13b1dc40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e5dd81990f445fac0f60cb13b1dc40" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 37819 } } }
I20260928 03:10:16.578550 29334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40 [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:16.578092 29335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62e5dd81990f445fac0f60cb13b1dc40. Latest consensus state: current_term: 1 leader_uuid: "62e5dd81990f445fac0f60cb13b1dc40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e5dd81990f445fac0f60cb13b1dc40" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 37819 } } }
I20260928 03:10:16.578884 29335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40 [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:16.584374 29340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:10:16.590639 29340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:10:16.600937 29340 catalog_manager.cc:1347] Generated new cluster ID: cd5cf6efcacf49f4aa48de2daf0f5815
I20260928 03:10:16.601171 29340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:10:16.630452 29340 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:10:16.630935 29340 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:10:16.640033 29340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62e5dd81990f445fac0f60cb13b1dc40: Generated new TSK 0
I20260928 03:10:16.641193 29340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:10:16.659920 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 29256
W20260928 03:10:16.661531 29318 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35877
W20260928 03:10:16.662510 6912 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 29256: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 03:10:16.662696 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 29256
2026-09-28T03:10:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2120 ms)
[----------] 18 tests from AuthTokenIssuingTest (37453 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:10:16 dist-test-slave-fjww krb5kdc[29354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:10:16 dist-test-slave-fjww krb5kdc[29354](info): set up 2 sockets
Sep 28 03:10:16 dist-test-slave-fjww krb5kdc[29354](info): commencing operation
krb5kdc: starting...
W20260928 03:10:17.808462 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:10:17 dist-test-slave-fjww krb5kdc[29354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:10:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:10:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:10:18.100101 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:38099
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:38099,127.6.192.61:33281,127.6.192.60:44575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:10:18.446710 29367 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 03:10:18.447064 29367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:10:18.447131 29367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:10:18.447216 29367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:10:18.454841 29367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:10:18.454955 29367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:10:18.455014 29367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:10:18.455065 29367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:10:18.455127 29367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:10:18.465400 29367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:38099,127.6.192.61:33281,127.6.192.60:44575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:38099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:10:18.466845 29367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:10:18.468200 29367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:10:18.480829 29372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:18.481097 29373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:18.481277 29375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:10:18.482182 29367 server_base.cc:1034] running on GCE node
I20260928 03:10:18.483070 29367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:10:18.484638 29367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:10:18.485908 29367 hybrid_clock.cc:648] HybridClock initialized: now 1790565018485868 us; error 63 us; skew 500 ppm
I20260928 03:10:18.486290 29367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:10:18 dist-test-slave-fjww krb5kdc[29354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565018, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:10:18.489859 29367 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:10:18.497386 29367 webserver.cc:460] Webserver started at http://127.6.192.62:40531/ using document root <none> and password file <none>
I20260928 03:10:18.498205 29367 fs_manager.cc:359] Metadata directory not provided
I20260928 03:10:18.498345 29367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:10:18.498747 29367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:10:18.502102 29367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "e418531319d74b5eb483f18bd598332d"
format_stamp: "Formatted at 2026-09-28 03:10:18 on dist-test-slave-fjww"
server_key: "f32ca4ea0c387e295f9e7b5d71bf988b"
server_key_iv: "161d53a3fa95d153d3ed5672805a4754"
server_key_version: "encryptionkey@0"
I20260928 03:10:18.502985 29367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "e418531319d74b5eb483f18bd598332d"
format_stamp: "Formatted at 2026-09-28 03:10:18 on dist-test-slave-fjww"
server_key: "f32ca4ea0c387e295f9e7b5d71bf988b"
server_key_iv: "161d53a3fa95d153d3ed5672805a4754"
server_key_version: "encryptionkey@0"
I20260928 03:10:18.509280 29367 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 03:10:18.513415 29382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:10:18.515163 29367 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 03:10:18.515386 29367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "e418531319d74b5eb483f18bd598332d"
format_stamp: "Formatted at 2026-09-28 03:10:18 on dist-test-slave-fjww"
server_key: "f32ca4ea0c387e295f9e7b5d71bf988b"
server_key_iv: "161d53a3fa95d153d3ed5672805a4754"
server_key_version: "encryptionkey@0"
I20260928 03:10:18.515690 29367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:10:18.536995 29367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:10:18.543277 29367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:10:18.543658 29367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:10:18.566757 29367 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:38099
I20260928 03:10:18.566802 29433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:38099 every 8 connection(s)
I20260928 03:10:18.568360 29367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:10:18.573724 29434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:10:18.578142 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 29367
I20260928 03:10:18.578377 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:10:18.579005 6912 external_mini_cluster.cc:1428] Setting key d9068ec02612540375b451775b95b2a1
I20260928 03:10:18.581247 29434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:10:18 dist-test-slave-fjww krb5kdc[29354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565018, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.192.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.61@KRBTEST.COM" created.
I20260928 03:10:18.608187 29436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:18.585040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54152 (local address 127.6.192.62:38099)
0928 03:10:18.586016 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:18.586034 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:18.586524 (+ 490us) server_negotiation.cc:407] Connection header received
0928 03:10:18.588914 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:18.588920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:18.589182 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:18.589424 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:18.590880 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:18.592305 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:18.593930 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:18.594541 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:18.594782 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:18.598518 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:18.598527 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:18.598540 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:18.598572 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:18.602412 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:18.603354 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:18.603372 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:18.603374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:18.603629 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:18.604237 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:18.604243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:18.604246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:18.605024 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:18.605270 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:18.605932 (+ 662us) server_negotiation.cc:299] Negotiation successful
0928 03:10:18.606353 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":654,"thread_start_us":411,"threads_started":1}
I20260928 03:10:18.608237 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:18.585040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:38099 (local address 127.0.0.1:54152)
0928 03:10:18.586350 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:18.586392 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 03:10:18.588470 (+ 2078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:10:18.589443 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:10:18.589451 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:10:18.590274 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:10:18.590639 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:18.590659 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:18.592479 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:18.592485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:18.593772 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:18.593785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:18.594810 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:18.594818 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:18.595006 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:18.595828 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:10:18.595853 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:10:18.598248 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:10:18.602587 (+ 4339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:18.602603 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:18.602618 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:18.603177 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:18.603816 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:18.603825 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:18.603830 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:18.604075 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:18.605208 (+ 1133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:10:18.605218 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:10:18.605478 (+ 260us) client_negotiation.cc:769] Sending connection context
0928 03:10:18.605787 (+ 309us) client_negotiation.cc:240] Negotiation successful
0928 03:10:18.606503 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"mutex_wait_us":181,"thread_start_us":793,"threads_started":1}
I20260928 03:10:18.612012 29434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:10:18.613440 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:18.612932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33281 (local address 127.0.0.1:54532)
0928 03:10:18.613145 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:18.613301 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260928 03:10:18.613897 29384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33281: connect: Connection refused (error 111)
I20260928 03:10:18.615744 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:18.615301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33281 (local address 127.0.0.1:54538)
0928 03:10:18.615508 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:18.615594 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 03:10:18.615995 29434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.61:33281: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33281: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.61.keytab.
Entry for principal kudu/127.6.192.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.61.keytab.
I20260928 03:10:18.638653 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.61:33281
--webserver_interface=127.6.192.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.61.keytab
--principal=kudu/127.6.192.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:38099,127.6.192.61:33281,127.6.192.60:44575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:10:18.665396 29434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } attempt: 1
I20260928 03:10:18.666836 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:18.666279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33281 (local address 127.0.0.1:54546)
0928 03:10:18.666528 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:18.666642 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 03:10:18.668118 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:18.667668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33281 (local address 127.0.0.1:54558)
0928 03:10:18.667847 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:18.667947 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 03:10:18.668367 29434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.61:33281: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33281: connect: Connection refused (error 111)
I20260928 03:10:18.736661 29434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } attempt: 2
I20260928 03:10:18.738057 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:18.737456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33281 (local address 127.0.0.1:54572)
0928 03:10:18.737749 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:18.737864 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 03:10:18.739553 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:18.738933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33281 (local address 127.0.0.1:54582)
0928 03:10:18.739133 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:18.739290 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 03:10:18.739861 29434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.61:33281: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33281: connect: Connection refused (error 111)
I20260928 03:10:18.831142 29434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } attempt: 3
I20260928 03:10:18.832650 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:18.832060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33281 (local address 127.0.0.1:54586)
0928 03:10:18.832323 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:18.832446 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 03:10:18.834239 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:18.833599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33281 (local address 127.0.0.1:54592)
0928 03:10:18.833875 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:18.834055 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260928 03:10:18.834550 29434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.61:33281: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33281: connect: Connection refused (error 111)
I20260928 03:10:18.977864 29434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } attempt: 4
I20260928 03:10:18.979130 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:18.978555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33281 (local address 127.0.0.1:54602)
0928 03:10:18.978892 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:18.978989 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260928 03:10:18.980255 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:18.979872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33281 (local address 127.0.0.1:54606)
0928 03:10:18.980047 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:18.980130 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 03:10:18.980476 29434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.61:33281: Network error: Client connection negotiation failed: client connection to 127.6.192.61:33281: connect: Connection refused (error 111)
W20260928 03:10:18.991264 29441 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 03:10:18.991865 29441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:10:18.991997 29441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:10:18.992123 29441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:10:18.999734 29441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:10:18.999868 29441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:10:19.000015 29441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:10:19.000104 29441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:10:19.000213 29441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:10:19.010529 29441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:38099,127.6.192.61:33281,127.6.192.60:44575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.61.keytab
--principal=kudu/127.6.192.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.61:33281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.192.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:10:19.012014 29441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:10:19.013413 29441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:10:19.026247 29449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:19.026310 29446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:19.026288 29447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:10:19.027199 29441 server_base.cc:1034] running on GCE node
I20260928 03:10:19.028180 29441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:10:19.029675 29441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:10:19.031014 29441 hybrid_clock.cc:648] HybridClock initialized: now 1790565019030935 us; error 52 us; skew 500 ppm
I20260928 03:10:19.031508 29441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:10:19 dist-test-slave-fjww krb5kdc[29354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565019, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:10:19.034854 29441 init.cc:377] Logged in from keytab as kudu/127.6.192.61@KRBTEST.COM (short username kudu)
I20260928 03:10:19.036437 29441 webserver.cc:460] Webserver started at http://127.6.192.61:39467/ using document root <none> and password file <none>
I20260928 03:10:19.037119 29441 fs_manager.cc:359] Metadata directory not provided
I20260928 03:10:19.037207 29441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:10:19.037484 29441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:10:19.040622 29441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-1/data/instance:
uuid: "461f91f7a7d74ad3ac5fca9004134383"
format_stamp: "Formatted at 2026-09-28 03:10:19 on dist-test-slave-fjww"
server_key: "5fffed7f835c2999697337a5ec56c1e5"
server_key_iv: "f5692106fb406f9cf5bcd866dc874dac"
server_key_version: "encryptionkey@0"
I20260928 03:10:19.041512 29441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-1/wal/instance:
uuid: "461f91f7a7d74ad3ac5fca9004134383"
format_stamp: "Formatted at 2026-09-28 03:10:19 on dist-test-slave-fjww"
server_key: "5fffed7f835c2999697337a5ec56c1e5"
server_key_iv: "f5692106fb406f9cf5bcd866dc874dac"
server_key_version: "encryptionkey@0"
I20260928 03:10:19.047254 29441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 03:10:19.050964 29456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:10:19.052454 29441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 03:10:19.052655 29441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-1/wal
uuid: "461f91f7a7d74ad3ac5fca9004134383"
format_stamp: "Formatted at 2026-09-28 03:10:19 on dist-test-slave-fjww"
server_key: "5fffed7f835c2999697337a5ec56c1e5"
server_key_iv: "f5692106fb406f9cf5bcd866dc874dac"
server_key_version: "encryptionkey@0"
I20260928 03:10:19.052983 29441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:10:19.066078 29441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:10:19.070763 29441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:10:19.071074 29441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:10:19.090672 29441 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.61:33281
I20260928 03:10:19.090679 29507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.61:33281 every 8 connection(s)
I20260928 03:10:19.092226 29441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-1/data/info.pb
I20260928 03:10:19.096174 29508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:10:19.096572 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 29441
I20260928 03:10:19.096741 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-1/wal/instance
I20260928 03:10:19.097172 6912 external_mini_cluster.cc:1428] Setting key 75d5c755a97603b343591d8fc67cebcf
I20260928 03:10:19.104632 29508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:10:19 dist-test-slave-fjww krb5kdc[29354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565019, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.61@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.192.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.60@KRBTEST.COM" created.
I20260928 03:10:19.128187 29436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.107389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54162 (local address 127.6.192.62:38099)
0928 03:10:19.107660 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:19.107669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:19.108935 (+ 1266us) server_negotiation.cc:407] Connection header received
0928 03:10:19.110910 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:19.110919 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:19.111018 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:19.111144 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:19.112925 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.114042 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.115779 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.116562 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.116735 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.120910 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:19.120917 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:19.120922 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:19.120969 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:19.124286 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.125560 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.125565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.125567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.125694 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.126278 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.126281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.126283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.126559 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:19.126671 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:19.127779 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0928 03:10:19.127961 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 03:10:19.129908 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.107902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:38099 (local address 127.0.0.1:54162)
0928 03:10:19.108744 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.108798 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 03:10:19.110609 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:10:19.111404 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:10:19.111416 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:10:19.112113 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:10:19.112721 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:19.112740 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.114222 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.114227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:19.115615 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:19.115629 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.116841 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.116847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:19.117098 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.117864 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:10:19.117889 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:10:19.120598 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:10:19.124480 (+ 3882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:19.124496 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:19.124521 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:19.125390 (+ 869us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:19.125828 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:19.125837 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:19.125844 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:19.126144 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:19.126704 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:10:19.126714 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:10:19.127303 (+ 589us) client_negotiation.cc:769] Sending connection context
0928 03:10:19.127670 (+ 367us) client_negotiation.cc:240] Negotiation successful
0928 03:10:19.128298 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":369,"threads_started":1}
I20260928 03:10:19.132027 29508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:10:19 dist-test-slave-fjww krb5kdc[29354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565019, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.61@KRBTEST.COM for kudu/127.6.192.61@KRBTEST.COM
I20260928 03:10:19.150291 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.132969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33281 (local address 127.0.0.1:54618)
0928 03:10:19.133234 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.133256 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 03:10:19.133380 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:10:19.134170 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:10:19.134183 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:10:19.134760 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:10:19.134945 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:19.134955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.136929 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.136939 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:19.138757 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:19.138777 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.139930 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.139940 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:19.140198 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.140906 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:10:19.140936 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:10:19.143938 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:10:19.147283 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:19.147290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:19.147294 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:19.147717 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:19.148084 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:19.148089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:19.148092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:19.148239 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:19.148887 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:10:19.148896 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:10:19.149124 (+ 228us) client_negotiation.cc:769] Sending connection context
0928 03:10:19.149332 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 03:10:19.149713 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 03:10:19.150318 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.133015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54618 (local address 127.6.192.61:33281)
0928 03:10:19.133702 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:19.133709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:19.133740 (+ 31us) server_negotiation.cc:407] Connection header received
0928 03:10:19.133790 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:19.133797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:19.133990 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:19.134117 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:19.135155 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.136765 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.138940 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.139678 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.139883 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.144160 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:19.144174 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:19.144189 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:19.144226 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:19.147133 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.147848 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.147863 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.147865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.147971 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.148339 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.148346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.148350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.148692 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:19.148938 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:19.149502 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 03:10:19.150003 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":263,"server-negotiator.queue_time_us":515,"thread_start_us":344,"threads_started":1}
I20260928 03:10:19.153128 29508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:10:19.154515 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.154020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37746)
0928 03:10:19.154224 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.154345 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260928 03:10:19.154855 29457 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
I20260928 03:10:19.156647 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.156277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37754)
0928 03:10:19.156443 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.156533 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 03:10:19.156908 29508 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:44575: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.60.keytab.
Entry for principal kudu/127.6.192.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.60.keytab.
I20260928 03:10:19.163750 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.60:44575
--webserver_interface=127.6.192.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.60.keytab
--principal=kudu/127.6.192.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:38099,127.6.192.61:33281,127.6.192.60:44575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:10:19.206322 29508 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } attempt: 1
I20260928 03:10:19.207669 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.207043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37758)
0928 03:10:19.207286 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.207462 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 03:10:19.208853 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.208444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37768)
0928 03:10:19.208607 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.208715 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 03:10:19.209108 29508 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:44575: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
I20260928 03:10:19.277390 29508 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } attempt: 2
I20260928 03:10:19.278606 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.278130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37784)
0928 03:10:19.278354 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.278445 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 03:10:19.279832 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.279287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37800)
0928 03:10:19.279423 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.279571 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 03:10:19.279752 29434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } attempt: 5
W20260928 03:10:19.280090 29508 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:44575: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Sep 28 03:10:19 dist-test-slave-fjww krb5kdc[29354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565018, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for kudu/127.6.192.61@KRBTEST.COM
I20260928 03:10:19.296244 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.280608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.6.192.61:33281)
0928 03:10:19.280793 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:19.280799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:19.280823 (+ 24us) server_negotiation.cc:407] Connection header received
0928 03:10:19.280884 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:19.280889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:19.280988 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:19.281109 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:19.282071 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.283172 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.284630 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.285266 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.285393 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.289009 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:19.289016 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:19.289021 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:19.289061 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:19.293212 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.294143 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.294149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.294152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.294292 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.294695 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.294700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.294706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.295015 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:19.295178 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:19.295692 (+ 514us) server_negotiation.cc:299] Negotiation successful
0928 03:10:19.295920 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 03:10:19.296253 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.280427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33281 (local address 127.0.0.1:54628)
0928 03:10:19.280633 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.280655 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 03:10:19.280781 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:10:19.281229 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:10:19.281245 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:10:19.281693 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:10:19.281904 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:19.281914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.283351 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.283356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:19.284459 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:19.284474 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.285463 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.285468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:19.285666 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.286444 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:10:19.286464 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:10:19.288776 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:10:19.293457 (+ 4681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:19.293462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:19.293465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:19.293961 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:19.294419 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:19.294425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:19.294435 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:19.294569 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:19.295141 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:10:19.295146 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:10:19.295364 (+ 218us) client_negotiation.cc:769] Sending connection context
0928 03:10:19.295570 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 03:10:19.295884 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 03:10:19.297546 29434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:10:19.298730 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.298276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37802)
0928 03:10:19.298498 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.298577 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 03:10:19.299800 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.299456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37818)
0928 03:10:19.299606 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.299689 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 03:10:19.300208 29434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:44575: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
I20260928 03:10:19.351516 29434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } attempt: 1
I20260928 03:10:19.352945 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.352268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37824)
0928 03:10:19.352491 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.352590 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 03:10:19.354355 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.353877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37826)
0928 03:10:19.354097 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.354183 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260928 03:10:19.354666 29434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:44575: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
I20260928 03:10:19.371384 29508 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } attempt: 3
I20260928 03:10:19.372781 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.372232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37832)
0928 03:10:19.372506 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.372606 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 03:10:19.374166 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.373675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37844)
0928 03:10:19.373950 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.374037 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
W20260928 03:10:19.374533 29508 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:44575: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
I20260928 03:10:19.422925 29434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } attempt: 2
I20260928 03:10:19.424353 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.423798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37856)
0928 03:10:19.424042 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.424136 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 03:10:19.425577 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.425164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37872)
0928 03:10:19.425375 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.425450 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 03:10:19.425858 29434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:44575: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
W20260928 03:10:19.514986 29515 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 03:10:19.515369 29515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:10:19.515537 29515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:10:19.515681 29515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 03:10:19.517788 29508 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } attempt: 4
I20260928 03:10:19.519202 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.518507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37876)
0928 03:10:19.518806 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.518914 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260928 03:10:19.520534 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.520167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37882)
0928 03:10:19.520313 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.520412 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 03:10:19.520807 29508 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:44575: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
W20260928 03:10:19.524202 29515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:10:19.524303 29515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:10:19.524385 29515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:10:19.524466 29515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:10:19.524559 29515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:10:19.532239 29434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } attempt: 3
I20260928 03:10:19.534124 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.533215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37884)
0928 03:10:19.533845 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.533952 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437}
I20260928 03:10:19.535406 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.534921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37900)
0928 03:10:19.535183 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.535278 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260928 03:10:19.535749 29434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:44575: Network error: Client connection negotiation failed: client connection to 127.6.192.60:44575: connect: Connection refused (error 111)
I20260928 03:10:19.536338 29515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:38099,127.6.192.61:33281,127.6.192.60:44575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.60.keytab
--principal=kudu/127.6.192.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.60:44575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.192.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:10:19.537597 29515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:10:19.538878 29515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:10:19.553603 29523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:19.553514 29521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:19.554177 29520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:10:19.554188 29515 server_base.cc:1034] running on GCE node
I20260928 03:10:19.555123 29515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:10:19.556788 29515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:10:19.558032 29515 hybrid_clock.cc:648] HybridClock initialized: now 1790565019557928 us; error 77 us; skew 500 ppm
I20260928 03:10:19.558441 29515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:10:19 dist-test-slave-fjww krb5kdc[29354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565019, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:10:19.561708 29515 init.cc:377] Logged in from keytab as kudu/127.6.192.60@KRBTEST.COM (short username kudu)
I20260928 03:10:19.563258 29515 webserver.cc:460] Webserver started at http://127.6.192.60:34519/ using document root <none> and password file <none>
I20260928 03:10:19.564034 29515 fs_manager.cc:359] Metadata directory not provided
I20260928 03:10:19.564175 29515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:10:19.564508 29515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:10:19.567806 29515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-2/data/instance:
uuid: "ead4691154af4f4aaeab43dc6d0c1e37"
format_stamp: "Formatted at 2026-09-28 03:10:19 on dist-test-slave-fjww"
server_key: "335b3e1984cb37aae7f5b9e327bc2836"
server_key_iv: "996f58d7cfd10e9a37f24a7143b80560"
server_key_version: "encryptionkey@0"
I20260928 03:10:19.568665 29515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-2/wal/instance:
uuid: "ead4691154af4f4aaeab43dc6d0c1e37"
format_stamp: "Formatted at 2026-09-28 03:10:19 on dist-test-slave-fjww"
server_key: "335b3e1984cb37aae7f5b9e327bc2836"
server_key_iv: "996f58d7cfd10e9a37f24a7143b80560"
server_key_version: "encryptionkey@0"
I20260928 03:10:19.574298 29515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:10:19.578037 29530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:10:19.579591 29515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 03:10:19.579784 29515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-2/wal
uuid: "ead4691154af4f4aaeab43dc6d0c1e37"
format_stamp: "Formatted at 2026-09-28 03:10:19 on dist-test-slave-fjww"
server_key: "335b3e1984cb37aae7f5b9e327bc2836"
server_key_iv: "996f58d7cfd10e9a37f24a7143b80560"
server_key_version: "encryptionkey@0"
I20260928 03:10:19.580029 29515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:10:19.609884 29515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:10:19.614617 29515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:10:19.614910 29515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:10:19.635411 29515 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.60:44575
I20260928 03:10:19.635504 29581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.60:44575 every 8 connection(s)
I20260928 03:10:19.636849 29515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-2/data/info.pb
I20260928 03:10:19.640766 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:10:19.641726 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 29515
I20260928 03:10:19.641919 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790564644636890-6912-0/minicluster-data/master-2/wal/instance
I20260928 03:10:19.642369 6912 external_mini_cluster.cc:1428] Setting key 19711433aee11d80cddf93c90d96021c
I20260928 03:10:19.650863 29582 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:10:19 dist-test-slave-fjww krb5kdc[29354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.60@KRBTEST.COM
Sep 28 03:10:19 dist-test-slave-fjww krb5kdc[29354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565019, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.60@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:10:19.669963 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.644387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37906 (local address 127.6.192.60:44575)
0928 03:10:19.645196 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:19.645212 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:19.645264 (+ 52us) server_negotiation.cc:407] Connection header received
0928 03:10:19.647617 (+ 2353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:19.647648 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:19.648220 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:19.648693 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:19.650052 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.652104 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.653671 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.655044 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.655246 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.660729 (+ 5483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:19.660737 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:19.660750 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:19.660799 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:19.664834 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.666001 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.666008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.666015 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.666176 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.666686 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.666689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.666692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.667254 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:19.667641 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:19.668124 (+ 483us) server_negotiation.cc:299] Negotiation successful
0928 03:10:19.668590 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":313,"threads_started":1}
I20260928 03:10:19.672186 29586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.653849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54164 (local address 127.6.192.62:38099)
0928 03:10:19.654485 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:19.654491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:19.655077 (+ 586us) server_negotiation.cc:407] Connection header received
0928 03:10:19.655282 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:19.655288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:19.655381 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:19.655539 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:19.657169 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.658467 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.660392 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.660940 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.661053 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.665515 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:19.665525 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:19.665532 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:19.665586 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:19.668949 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.669849 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.669854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.669857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.670003 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.670508 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.670513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.670515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.670808 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:19.670974 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:19.671513 (+ 539us) server_negotiation.cc:299] Negotiation successful
0928 03:10:19.671756 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":298,"threads_started":1}
I20260928 03:10:19.672315 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.653869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:38099 (local address 127.0.0.1:54164)
0928 03:10:19.654908 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.654964 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 03:10:19.655142 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:10:19.655611 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:10:19.655623 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:10:19.656628 (+ 1005us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:10:19.656999 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:19.657009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.658696 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.658701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:19.660201 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:19.660214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.661158 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.661167 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:19.661394 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.662165 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:10:19.662196 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:10:19.665249 (+ 3053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:10:19.669096 (+ 3847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:19.669105 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:19.669112 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:19.669676 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:19.670145 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:19.670162 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:19.670169 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:19.670368 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:19.670948 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:10:19.670956 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:10:19.671161 (+ 205us) client_negotiation.cc:769] Sending connection context
0928 03:10:19.671372 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 03:10:19.671862 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":601,"threads_started":1}
I20260928 03:10:19.673810 29582 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:10:19 dist-test-slave-fjww krb5kdc[29354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565019, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.60@KRBTEST.COM for kudu/127.6.192.61@KRBTEST.COM
Sep 28 03:10:19 dist-test-slave-fjww krb5kdc[29354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:10:19.689036 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.674606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:33281 (local address 127.0.0.1:54638)
0928 03:10:19.674889 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.674906 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 03:10:19.675041 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:10:19.675405 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:10:19.675410 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:10:19.675844 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:10:19.675995 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:19.676004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.677272 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.677281 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:19.678295 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:19.678308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.679196 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.679205 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:19.679413 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.680061 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:10:19.680087 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:10:19.682748 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:10:19.686268 (+ 3520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:19.686273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:19.686278 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:19.686666 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:19.687090 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:19.687096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:19.687099 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:19.687307 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:19.688189 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:10:19.688196 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:10:19.688383 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 03:10:19.688562 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 03:10:19.688789 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 03:10:19.689628 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.674615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54638 (local address 127.6.192.61:33281)
0928 03:10:19.674804 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:19.674811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:19.674981 (+ 170us) server_negotiation.cc:407] Connection header received
0928 03:10:19.675164 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:19.675170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:19.675269 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:19.675381 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:19.676150 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.677117 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.678466 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.679023 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.679169 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.682951 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:19.682956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:19.682960 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:19.682993 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:19.686125 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.686790 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.686797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.686801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.686966 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.687517 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.687522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.687525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.688026 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:19.688224 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:19.688695 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 03:10:19.688935 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 03:10:19.690418 29586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.673770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54180 (local address 127.6.192.62:38099)
0928 03:10:19.673927 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:19.673931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:19.674003 (+ 72us) server_negotiation.cc:407] Connection header received
0928 03:10:19.674221 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:19.674226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:19.674320 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:19.674461 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:19.675830 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.676530 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.677758 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.678205 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.678344 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.683875 (+ 5531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:19.683883 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:19.683888 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:19.683928 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:19.687091 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.687915 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.687921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.687925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.688099 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.688598 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.688604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.688607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.689000 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:19.689330 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:19.689689 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 03:10:19.690069 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 03:10:19.690719 29582 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:10:19 dist-test-slave-fjww krb5kdc[29354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565019, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.60@KRBTEST.COM for kudu/127.6.192.60@KRBTEST.COM
Sep 28 03:10:19 dist-test-slave-fjww krb5kdc[29354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.61@KRBTEST.COM
I20260928 03:10:19.705919 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.691633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37914 (local address 127.6.192.60:44575)
0928 03:10:19.691823 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:19.691828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:19.692029 (+ 201us) server_negotiation.cc:407] Connection header received
0928 03:10:19.692158 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:19.692166 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:19.692230 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:19.692339 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:19.693515 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.694455 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.696099 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.696542 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.696659 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.700371 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:19.700376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:19.700379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:19.700408 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:19.703314 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.703974 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.703985 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.703989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.704095 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.704449 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.704453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.704456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.704732 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:19.704880 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:19.705236 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 03:10:19.705614 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":149,"server-negotiator.queue_time_us":67}
I20260928 03:10:19.705919 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.691633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37914)
0928 03:10:19.691930 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.691948 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:10:19.692048 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:10:19.692719 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:10:19.692723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:10:19.693141 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:10:19.693349 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:19.693361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.694598 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.694602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:19.695944 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:19.695960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.696726 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.696736 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:19.696932 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.697584 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:10:19.697615 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:10:19.700186 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:10:19.703474 (+ 3288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:19.703482 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:19.703488 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:19.703850 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:19.704201 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:19.704209 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:19.704213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:19.704341 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:19.704844 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:10:19.704848 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:10:19.704993 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 03:10:19.705132 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 03:10:19.705464 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260928 03:10:19.708115 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.692584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.6.192.61:33281)
0928 03:10:19.692765 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:19.692769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:19.692786 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:10:19.692835 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:19.692839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:19.692904 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:19.692988 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:19.694407 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.695245 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.696758 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.697457 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.697578 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.702427 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:19.702433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:19.702436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:19.702471 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:19.705125 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.705830 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.705837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.705840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.705994 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.706642 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.706646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.706648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.706971 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:19.707265 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:19.707531 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 03:10:19.707810 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 03:10:19.712324 29582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37: Bootstrap starting.
I20260928 03:10:19.712980 29434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } attempt: 4
I20260928 03:10:19.714229 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:10:19.716691 29582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37: Neither blocks nor log segments found. Creating new log.
I20260928 03:10:19.718257 29582 log.cc:824] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37: Log is configured to *not* fsync() on all Append() calls
I20260928 03:10:19.717792 29409 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e418531319d74b5eb483f18bd598332d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54180
I20260928 03:10:19.722282 29582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37: No bootstrap required, opened a new log
Sep 28 03:10:19 dist-test-slave-fjww krb5kdc[29354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565018, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for kudu/127.6.192.60@KRBTEST.COM
I20260928 03:10:19.730895 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.713588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37922)
0928 03:10:19.713872 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.713896 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 03:10:19.714070 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:10:19.714454 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:10:19.714459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:10:19.714946 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:10:19.715175 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:19.715187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.716806 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.716813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:19.718604 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:19.718622 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.719916 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.719923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:19.720117 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.720820 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:10:19.720843 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:10:19.723963 (+ 3120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:10:19.727916 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:19.727923 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:19.727928 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:19.728612 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:19.729066 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:19.729072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:19.729076 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:19.729226 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:19.729809 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:10:19.729814 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:10:19.729991 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 03:10:19.730142 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 03:10:19.730502 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 03:10:19.730892 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.713862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37922 (local address 127.6.192.60:44575)
0928 03:10:19.714105 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:19.714111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:19.714133 (+ 22us) server_negotiation.cc:407] Connection header received
0928 03:10:19.714189 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:19.714193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:19.714276 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:19.715333 (+ 1057us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:19.715510 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.716612 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.718790 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.719680 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.719885 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.724206 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:19.724213 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:19.724218 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:19.724264 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:19.727770 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.728759 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.728764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.728767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.728907 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.729345 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.729350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.729357 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.729655 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:19.729831 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:19.730236 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 03:10:19.730480 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 03:10:19.734282 29582 raft_consensus.cc:348] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e418531319d74b5eb483f18bd598332d" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } } peers { permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } } peers { permanent_uuid: "ead4691154af4f4aaeab43dc6d0c1e37" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } }
I20260928 03:10:19.734822 29582 raft_consensus.cc:729] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ead4691154af4f4aaeab43dc6d0c1e37, State: Initialized, Role: FOLLOWER
I20260928 03:10:19.735389 29582 consensus_queue.cc:260] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e418531319d74b5eb483f18bd598332d" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } } peers { permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } } peers { permanent_uuid: "ead4691154af4f4aaeab43dc6d0c1e37" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } }
I20260928 03:10:19.736240 29434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d: Bootstrap starting.
I20260928 03:10:19.736515 29589 sys_catalog.cc:455] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e418531319d74b5eb483f18bd598332d" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } } peers { permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } } peers { permanent_uuid: "ead4691154af4f4aaeab43dc6d0c1e37" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } } }
I20260928 03:10:19.736967 29589 sys_catalog.cc:458] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:10:19.737866 29582 sys_catalog.cc:564] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:10:19.739749 29434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d: Neither blocks nor log segments found. Creating new log.
I20260928 03:10:19.740931 29434 log.cc:824] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d: Log is configured to *not* fsync() on all Append() calls
I20260928 03:10:19.744508 29434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d: No bootstrap required, opened a new log
W20260928 03:10:19.751356 29601 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 03:10:19.751663 29601 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 03:10:19.752862 29434 raft_consensus.cc:348] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e418531319d74b5eb483f18bd598332d" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } } peers { permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } } peers { permanent_uuid: "ead4691154af4f4aaeab43dc6d0c1e37" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } }
I20260928 03:10:19.753320 29434 raft_consensus.cc:729] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e418531319d74b5eb483f18bd598332d, State: Initialized, Role: FOLLOWER
I20260928 03:10:19.753876 29434 consensus_queue.cc:260] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e418531319d74b5eb483f18bd598332d" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } } peers { permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } } peers { permanent_uuid: "ead4691154af4f4aaeab43dc6d0c1e37" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } }
I20260928 03:10:19.755357 29603 sys_catalog.cc:455] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e418531319d74b5eb483f18bd598332d" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } } peers { permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } } peers { permanent_uuid: "ead4691154af4f4aaeab43dc6d0c1e37" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } } }
I20260928 03:10:19.756258 29603 sys_catalog.cc:458] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:10:19.756664 29434 sys_catalog.cc:564] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:10:19.756748 29409 raft_consensus.cc:482] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 03:10:19.756973 29409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:10:19.760815 29409 raft_consensus.cc:504] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e418531319d74b5eb483f18bd598332d" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } } peers { permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } } peers { permanent_uuid: "ead4691154af4f4aaeab43dc6d0c1e37" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } }
I20260928 03:10:19.747289 29515 master_runner.cc:186] Error getting master registration for 127.6.192.62:38099: OK, instance_id { permanent_uuid: "e418531319d74b5eb483f18bd598332d" instance_seqno: 1790565018560645 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 03:10:19.762110 29515 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 03:10:19.763092 29409 leader_election.cc:288] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [CANDIDATE]: Term 1 election: Requested vote from peers 461f91f7a7d74ad3ac5fca9004134383 (127.6.192.61:33281), ead4691154af4f4aaeab43dc6d0c1e37 (127.6.192.60:44575)
I20260928 03:10:19.764112 29557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e418531319d74b5eb483f18bd598332d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ead4691154af4f4aaeab43dc6d0c1e37"
I20260928 03:10:19.764653 29557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:10:19.769603 29557 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e418531319d74b5eb483f18bd598332d in term 1.
I20260928 03:10:19.770608 29383 leader_election.cc:302] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e418531319d74b5eb483f18bd598332d, ead4691154af4f4aaeab43dc6d0c1e37; no voters:
I20260928 03:10:19.771413 29603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:10:19.772328 29603 raft_consensus.cc:686] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [term 1 LEADER]: Becoming Leader. State: Replica: e418531319d74b5eb483f18bd598332d, State: Running, Role: LEADER
I20260928 03:10:19.773116 29603 consensus_queue.cc:237] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e418531319d74b5eb483f18bd598332d" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } } peers { permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } } peers { permanent_uuid: "ead4691154af4f4aaeab43dc6d0c1e37" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } }
I20260928 03:10:19.777575 29609 sys_catalog.cc:455] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e418531319d74b5eb483f18bd598332d. Latest consensus state: current_term: 1 leader_uuid: "e418531319d74b5eb483f18bd598332d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e418531319d74b5eb483f18bd598332d" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } } peers { permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } } peers { permanent_uuid: "ead4691154af4f4aaeab43dc6d0c1e37" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } } }
I20260928 03:10:19.778100 29609 sys_catalog.cc:458] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:19.795768 29613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260928 03:10:19.797230 29388 tablet.cc:2369] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:10:19.811844 29613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:10:19.820030 29508 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } attempt: 5
I20260928 03:10:19.846805 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.778657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54648 (local address 127.6.192.61:33281)
0928 03:10:19.778854 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:19.778863 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:19.800057 (+ 21194us) server_negotiation.cc:407] Connection header received
0928 03:10:19.800920 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:19.800928 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:19.801024 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:19.801102 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:19.803453 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.804638 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.808158 (+ 3520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.808927 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.809049 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.816586 (+ 7537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:19.816593 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:19.816599 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:19.816654 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:19.820722 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.840282 (+ 19560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.840292 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.840299 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.840585 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.842597 (+ 2012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.842606 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.842612 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.843163 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:19.843515 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:19.845767 (+ 2252us) server_negotiation.cc:299] Negotiation successful
0928 03:10:19.846184 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 03:10:19 dist-test-slave-fjww krb5kdc[29354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565019, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.61@KRBTEST.COM for kudu/127.6.192.60@KRBTEST.COM
I20260928 03:10:19.859192 29557 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37 [term 1 FOLLOWER]: Refusing update from remote peer e418531319d74b5eb483f18bd598332d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:10:19.860586 29609 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ead4691154af4f4aaeab43dc6d0c1e37" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:10:19.862190 29586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.773173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54196 (local address 127.6.192.62:38099)
0928 03:10:19.773389 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:19.773402 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:19.797515 (+ 24113us) server_negotiation.cc:407] Connection header received
0928 03:10:19.797748 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:19.797753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:19.797839 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:19.797963 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:19.804964 (+ 7001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.806128 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.815591 (+ 9463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.816566 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.816667 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.832686 (+ 16019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:19.832699 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:19.832708 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:19.832852 (+ 144us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:19.852002 (+ 19150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.857952 (+ 5950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.857960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.857964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.858199 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.859154 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.859168 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.859174 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.859747 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:19.860135 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:19.861518 (+ 1383us) server_negotiation.cc:299] Negotiation successful
0928 03:10:19.861796 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 03:10:19.867252 29536 tablet.cc:2369] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:10:19.871289 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.787368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37924 (local address 127.6.192.60:44575)
0928 03:10:19.787627 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:19.787636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:19.801312 (+ 13676us) server_negotiation.cc:407] Connection header received
0928 03:10:19.802207 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:19.802215 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:19.802320 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:19.802399 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:19.812506 (+ 10107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.813651 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.823092 (+ 9441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.824145 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.824308 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.849719 (+ 25411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:19.849730 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:19.849738 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:19.849805 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:19.855288 (+ 5483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.864999 (+ 9711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.865007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.865010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.865460 (+ 450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.866587 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.866597 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.866603 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.867572 (+ 969us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:19.868186 (+ 614us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:19.869503 (+ 1317us) server_negotiation.cc:299] Negotiation successful
0928 03:10:19.870233 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 03:10:19.885310 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.820887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:44575 (local address 127.0.0.1:37936)
0928 03:10:19.821487 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:19.821510 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 03:10:19.821627 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:10:19.838535 (+ 16908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:10:19.838542 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:10:19.839146 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:10:19.839460 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:19.839479 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.842038 (+ 2559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.842045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:19.843723 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:19.843742 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.846073 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.846081 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:19.846388 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.847241 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:10:19.847270 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:10:19.850671 (+ 3401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:10:19.880474 (+ 29803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:19.880484 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:19.880492 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:19.881295 (+ 803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:19.882142 (+ 847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:19.882149 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:19.882154 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:19.882390 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:19.883908 (+ 1518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:10:19.883926 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:10:19.884155 (+ 229us) client_negotiation.cc:769] Sending connection context
0928 03:10:19.884280 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 03:10:19.884716 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457}
I20260928 03:10:19.888388 29619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:19.822254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.6.192.60:44575)
0928 03:10:19.837929 (+ 15675us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:19.837937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:19.837974 (+ 37us) server_negotiation.cc:407] Connection header received
0928 03:10:19.838053 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:19.838059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:19.838201 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:19.839687 (+ 1486us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:19.840038 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.841745 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.844013 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:19.844828 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:19.850966 (+ 6138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:19.851263 (+ 297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:19.851271 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:19.851276 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:19.851312 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:19.880123 (+ 28811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.881594 (+ 1471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.881601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.881605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.881928 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:19.882669 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:19.882678 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:19.882684 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:19.883633 (+ 949us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:19.886109 (+ 2476us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:19.886439 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 03:10:19.887048 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15467,"thread_start_us":15096,"threads_started":1}
I20260928 03:10:19.888872 29629 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 03:10:19.896788 29508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383: Bootstrap starting.
I20260928 03:10:19.909407 29508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383: Neither blocks nor log segments found. Creating new log.
I20260928 03:10:19.898399 29367 master_runner.cc:186] Error getting master registration for 127.6.192.61:33281: OK, instance_id { permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" instance_seqno: 1790565019085121 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 03:10:19.910398 29367 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 03:10:19.911124 29508 log.cc:824] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383: Log is configured to *not* fsync() on all Append() calls
I20260928 03:10:19.917358 29589 sys_catalog.cc:455] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e418531319d74b5eb483f18bd598332d. Latest consensus state: current_term: 1 leader_uuid: "e418531319d74b5eb483f18bd598332d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e418531319d74b5eb483f18bd598332d" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } } peers { permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } } peers { permanent_uuid: "ead4691154af4f4aaeab43dc6d0c1e37" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } } }
I20260928 03:10:19.917752 29589 sys_catalog.cc:458] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:10:19.918385 29618 mvcc.cc:205] Tried to move back new op lower bound from 7334154321327382528 to 7334154320997584896. Current Snapshot: MvccSnapshot[applied={T|T < 7334154321327382528}]
I20260928 03:10:19.940397 29589 sys_catalog.cc:455] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e418531319d74b5eb483f18bd598332d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e418531319d74b5eb483f18bd598332d" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } } peers { permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } } peers { permanent_uuid: "ead4691154af4f4aaeab43dc6d0c1e37" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } } }
I20260928 03:10:19.940683 29589 sys_catalog.cc:458] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:10:19.944766 29508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383: No bootstrap required, opened a new log
I20260928 03:10:19.945412 29603 sys_catalog.cc:455] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e418531319d74b5eb483f18bd598332d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e418531319d74b5eb483f18bd598332d" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } } peers { permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } } peers { permanent_uuid: "ead4691154af4f4aaeab43dc6d0c1e37" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } } }
I20260928 03:10:19.945766 29603 sys_catalog.cc:458] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:19.946381 29613 catalog_manager.cc:1347] Generated new cluster ID: 648bd000ece949748c1ee925a12e010d
I20260928 03:10:19.946501 29613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:10:19.956202 29508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e418531319d74b5eb483f18bd598332d" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } } peers { permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } } peers { permanent_uuid: "ead4691154af4f4aaeab43dc6d0c1e37" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } }
I20260928 03:10:19.956698 29508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 461f91f7a7d74ad3ac5fca9004134383, State: Initialized, Role: FOLLOWER
I20260928 03:10:19.957396 29508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e418531319d74b5eb483f18bd598332d" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } } peers { permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } } peers { permanent_uuid: "ead4691154af4f4aaeab43dc6d0c1e37" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } }
I20260928 03:10:19.960229 29508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:10:19.962054 29483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e418531319d74b5eb483f18bd598332d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "461f91f7a7d74ad3ac5fca9004134383"
I20260928 03:10:19.961994 29482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383 [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:10:19.968189 29482 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383 [term 1 FOLLOWER]: Refusing update from remote peer e418531319d74b5eb483f18bd598332d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:10:19.972290 29483 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e418531319d74b5eb483f18bd598332d in term 1.
I20260928 03:10:19.978184 29609 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [LEADER]: Connected to new peer: Peer: permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:10:19.958984 29633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e418531319d74b5eb483f18bd598332d" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } } peers { permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } } peers { permanent_uuid: "ead4691154af4f4aaeab43dc6d0c1e37" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } } }
I20260928 03:10:19.980401 29633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:10:19.992928 29613 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:10:19.995115 29613 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:10:20.013499 29613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d: Generated new TSK 0
I20260928 03:10:20.014974 29613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:10:20.023555 29633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e418531319d74b5eb483f18bd598332d. Latest consensus state: current_term: 1 leader_uuid: "e418531319d74b5eb483f18bd598332d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e418531319d74b5eb483f18bd598332d" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } } peers { permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } } peers { permanent_uuid: "ead4691154af4f4aaeab43dc6d0c1e37" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } } }
I20260928 03:10:20.023968 29633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:10:20.025271 29646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e418531319d74b5eb483f18bd598332d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e418531319d74b5eb483f18bd598332d" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } } peers { permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } } peers { permanent_uuid: "ead4691154af4f4aaeab43dc6d0c1e37" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } } }
I20260928 03:10:20.025745 29646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:10:20.027365 29603 sys_catalog.cc:455] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e418531319d74b5eb483f18bd598332d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e418531319d74b5eb483f18bd598332d" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 38099 } } peers { permanent_uuid: "461f91f7a7d74ad3ac5fca9004134383" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 33281 } } peers { permanent_uuid: "ead4691154af4f4aaeab43dc6d0c1e37" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 44575 } } }
I20260928 03:10:20.027889 29603 sys_catalog.cc:458] T 00000000000000000000000000000000 P e418531319d74b5eb483f18bd598332d [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:20.082846 29648 catalog_manager.cc:1259] Loaded cluster ID: 648bd000ece949748c1ee925a12e010d
I20260928 03:10:20.083055 29648 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383: loading cluster ID for follower catalog manager: success
I20260928 03:10:20.090480 29648 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383: acquiring CA information for follower catalog manager: success
I20260928 03:10:20.101994 29648 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 461f91f7a7d74ad3ac5fca9004134383: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 03:10:20.198112 29586 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:20.178392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.6.192.62:38099)
0928 03:10:20.178624 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.178630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.195865 (+ 17235us) server_negotiation.cc:407] Connection header received
0928 03:10:20.196393 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.196400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.196500 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.196580 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:20.197829 (+ 1249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54202: BlockingRecv error: recv got EOF from 127.0.0.1:54202 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 03:10:20.201787 29619 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:20.190244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37938 (local address 127.6.192.60:44575)
0928 03:10:20.190436 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.190442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.200140 (+ 9698us) server_negotiation.cc:407] Connection header received
0928 03:10:20.200391 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.200397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.200515 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.200593 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:20.201521 (+ 928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37938: BlockingRecv error: recv got EOF from 127.0.0.1:37938 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 03:10:20.204030 29512 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:20.180144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.6.192.61:33281)
0928 03:10:20.180323 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.180329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.196042 (+ 15713us) server_negotiation.cc:407] Connection header received
0928 03:10:20.196388 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.196395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.196500 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.196581 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:20.203741 (+ 7160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54660: BlockingRecv error: recv got EOF from 127.0.0.1:54660 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 03:10:20.264887 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:20.247166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54666 (local address 127.6.192.61:33281)
0928 03:10:20.247406 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.247413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.256120 (+ 8707us) server_negotiation.cc:407] Connection header received
0928 03:10:20.256412 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.256419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.256684 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.256835 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:10:20.257938 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.259111 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.261333 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.262077 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.262189 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:20.263139 (+ 950us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:10:20.263956 (+ 817us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:10:20.264094 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:20.264389 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 03:10:20.264495 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260928 03:10:20.316636 29619 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:20.299045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37952 (local address 127.6.192.60:44575)
0928 03:10:20.299270 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.299279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.314228 (+ 14949us) server_negotiation.cc:407] Connection header received
0928 03:10:20.314873 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.314886 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.315045 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.315130 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:20.316341 (+ 1211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37952: BlockingRecv error: recv got EOF from 127.0.0.1:37952 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 03:10:20.368213 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:20.349199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54668 (local address 127.6.192.61:33281)
0928 03:10:20.349420 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.349431 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.360546 (+ 11115us) server_negotiation.cc:407] Connection header received
0928 03:10:20.361200 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.361207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.361310 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.361383 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:10:20.362173 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.363377 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.365608 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.366371 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.366486 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:20.367211 (+ 725us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:10:20.367406 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:10:20.367503 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:20.367769 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 03:10:20.367869 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260928 03:10:20.423259 29619 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:20.410066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37956 (local address 127.6.192.60:44575)
0928 03:10:20.410315 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.410322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.420999 (+ 10677us) server_negotiation.cc:407] Connection header received
0928 03:10:20.421599 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.421605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.421710 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.421798 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:20.422993 (+ 1195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37956: BlockingRecv error: recv got EOF from 127.0.0.1:37956 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 03:10:20.479763 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:20.461446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54682 (local address 127.6.192.61:33281)
0928 03:10:20.461650 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.461659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.472468 (+ 10809us) server_negotiation.cc:407] Connection header received
0928 03:10:20.473033 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.473040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.473134 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.473201 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:10:20.473925 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.475093 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.477230 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.477954 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.478082 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:20.478771 (+ 689us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:10:20.478940 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:10:20.479007 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:20.479259 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 03:10:20.479363 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 03:10:20.526714 29619 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:20.514590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37970 (local address 127.6.192.60:44575)
0928 03:10:20.514824 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.514832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.524668 (+ 9836us) server_negotiation.cc:407] Connection header received
0928 03:10:20.525181 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.525187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.525294 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.525366 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:20.526406 (+ 1040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37970: BlockingRecv error: recv got EOF from 127.0.0.1:37970 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 03:10:20.559662 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:20.552760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54684 (local address 127.6.192.61:33281)
0928 03:10:20.552953 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.552957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.553652 (+ 695us) server_negotiation.cc:407] Connection header received
0928 03:10:20.553883 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.553889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.553977 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.554081 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:10:20.554698 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.555844 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.557525 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.558141 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.558446 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:20.558860 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:10:20.558992 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:10:20.559099 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:20.559248 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 03:10:20.559330 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 03:10:20.569947 29619 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:20.567175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37984 (local address 127.6.192.60:44575)
0928 03:10:20.567465 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.567473 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.568368 (+ 895us) server_negotiation.cc:407] Connection header received
0928 03:10:20.568608 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.568614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.568703 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.568800 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:20.569720 (+ 920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37984: BlockingRecv error: recv got EOF from 127.0.0.1:37984 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
I20260928 03:10:20.591102 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:20.582639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54686 (local address 127.6.192.61:33281)
0928 03:10:20.582814 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.582820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.586115 (+ 3295us) server_negotiation.cc:407] Connection header received
0928 03:10:20.586349 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.586355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.586440 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.586544 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:10:20.586955 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.587864 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.589184 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.589684 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.589847 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:20.590276 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:10:20.590390 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:10:20.590538 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:20.590696 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 03:10:20.590779 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 03:10:20.599737 29619 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:20.597712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38000 (local address 127.6.192.60:44575)
0928 03:10:20.597938 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.597944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.598278 (+ 334us) server_negotiation.cc:407] Connection header received
0928 03:10:20.598508 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.598512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.598582 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.598711 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:20.599479 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38000: BlockingRecv error: recv got EOF from 127.0.0.1:38000 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 03:10:20.626331 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:20.619932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54690 (local address 127.6.192.61:33281)
0928 03:10:20.620158 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.620164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.620503 (+ 339us) server_negotiation.cc:407] Connection header received
0928 03:10:20.620666 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.620672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.620794 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.620914 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:10:20.621430 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.622703 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.624271 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.624985 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.625121 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:20.625540 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:10:20.625688 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:10:20.625795 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:20.625924 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 03:10:20.626016 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 03:10:20.636234 29619 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:20.633521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38006 (local address 127.6.192.60:44575)
0928 03:10:20.633737 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.633744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.634346 (+ 602us) server_negotiation.cc:407] Connection header received
0928 03:10:20.634599 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.634604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.634694 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.634829 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:20.635960 (+ 1131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38006: BlockingRecv error: recv got EOF from 127.0.0.1:38006 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 03:10:20.682838 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:20.677484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54698 (local address 127.6.192.61:33281)
0928 03:10:20.677811 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.677817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.678080 (+ 263us) server_negotiation.cc:407] Connection header received
0928 03:10:20.678368 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.678373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.678441 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.678545 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:10:20.678993 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.679808 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.681125 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.681551 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.681688 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:20.682098 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:10:20.682212 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:10:20.682311 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:20.682466 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 03:10:20.682547 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 03:10:20.690654 29619 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:20.688687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.6.192.60:44575)
0928 03:10:20.688939 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.688947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.689273 (+ 326us) server_negotiation.cc:407] Connection header received
0928 03:10:20.689511 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.689516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.689583 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.689685 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:20.690395 (+ 710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38022: BlockingRecv error: recv got EOF from 127.0.0.1:38022 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 03:10:20.753808 29601 catalog_manager.cc:1259] Loaded cluster ID: 648bd000ece949748c1ee925a12e010d
I20260928 03:10:20.753919 29601 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37: loading cluster ID for follower catalog manager: success
I20260928 03:10:20.757102 29601 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37: acquiring CA information for follower catalog manager: success
I20260928 03:10:20.760099 29601 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ead4691154af4f4aaeab43dc6d0c1e37: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 03:10:20.764410 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:20.758754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54702 (local address 127.6.192.61:33281)
0928 03:10:20.758935 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.758940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.759464 (+ 524us) server_negotiation.cc:407] Connection header received
0928 03:10:20.759707 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.759713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.759828 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.759939 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:10:20.760329 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.761076 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.762523 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.763012 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.763166 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:20.763613 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:10:20.763839 (+ 226us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:10:20.763923 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:20.764109 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 03:10:20.764182 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 03:10:20.777216 29619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:20.770477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38024 (local address 127.6.192.60:44575)
0928 03:10:20.770735 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.770744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.771258 (+ 514us) server_negotiation.cc:407] Connection header received
0928 03:10:20.771589 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.771594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.771768 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.771919 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:10:20.772382 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.773165 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.774643 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.775128 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.775286 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:20.775924 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:10:20.776506 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:10:20.776684 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:20.776793 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 03:10:20.776890 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 03:10:20.796344 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:20.784695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54708 (local address 127.6.192.61:33281)
0928 03:10:20.784983 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.784993 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.789166 (+ 4173us) server_negotiation.cc:407] Connection header received
0928 03:10:20.789478 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.789483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.789593 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.789685 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:10:20.790681 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.791674 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.793980 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.794629 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.794742 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:20.795407 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:10:20.795587 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:10:20.795657 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:20.795883 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 03:10:20.796003 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 03:10:20.797686 29744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:20.784349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54210 (local address 127.6.192.62:38099)
0928 03:10:20.788180 (+ 3831us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.788188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.789422 (+ 1234us) server_negotiation.cc:407] Connection header received
0928 03:10:20.789582 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.789587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.789810 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.790023 (+ 213us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:10:20.790853 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.792046 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.794322 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.794880 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.795014 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:20.795831 (+ 817us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:10:20.796496 (+ 665us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:10:20.796669 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:20.797190 (+ 521us) server_negotiation.cc:299] Negotiation successful
0928 03:10:20.797304 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3661,"thread_start_us":3526,"threads_started":1}
I20260928 03:10:20.800848 29619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:20.784472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38036 (local address 127.6.192.60:44575)
0928 03:10:20.784667 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:20.784673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:20.789282 (+ 4609us) server_negotiation.cc:407] Connection header received
0928 03:10:20.789588 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:20.789593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:20.789687 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:20.789774 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:10:20.790904 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.792041 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.793833 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:20.794531 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:20.794677 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:20.795297 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:10:20.795542 (+ 245us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:10:20.795638 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:20.800436 (+ 4798us) server_negotiation.cc:299] Negotiation successful
0928 03:10:20.800537 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 03:10:20.805742 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 29367
I20260928 03:10:20.807369 29429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:10:20.918939 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 29367
W20260928 03:10:20.939492 29460 connection.cc:535] server connection from 127.0.0.1:54628 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 03:10:20.940141 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 29441
I20260928 03:10:20.941269 29503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:10:21.051031 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 29441
I20260928 03:10:21.076675 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 29515
I20260928 03:10:21.077900 29577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:10:21.185359 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 29515
2026-09-28T03:10:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4527 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 03:10:21 dist-test-slave-fjww krb5kdc[29753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 03:10:21 dist-test-slave-fjww krb5kdc[29753](info): set up 2 sockets
Sep 28 03:10:21 dist-test-slave-fjww krb5kdc[29753](info): commencing operation
krb5kdc: starting...
W20260928 03:10:22.308300 6912 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/test-user.keytab.
Sep 28 03:10:22 dist-test-slave-fjww krb5kdc[29753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T03:10:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T03:10:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.192.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
Entry for principal kudu/127.6.192.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab.
I20260928 03:10:22.600145 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:46039
--webserver_interface=127.6.192.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:46039,127.6.192.61:43865,127.6.192.60:40075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 03:10:22.976723 29766 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 03:10:22.977156 29766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:10:22.977305 29766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:10:22.977458 29766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:10:22.986069 29766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:10:22.986196 29766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:10:22.986352 29766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:10:22.986441 29766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:10:22.986570 29766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:10:22.998150 29766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:46039,127.6.192.61:43865,127.6.192.60:40075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.62.keytab
--principal=kudu/127.6.192.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.62:46039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.192.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:10:22.999785 29766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:10:23.001320 29766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:10:23.015305 29771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:23.015270 29772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:23.015357 29774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:10:23.016162 29766 server_base.cc:1034] running on GCE node
I20260928 03:10:23.017104 29766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:10:23.018841 29766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:10:23.020136 29766 hybrid_clock.cc:648] HybridClock initialized: now 1790565023020045 us; error 61 us; skew 500 ppm
I20260928 03:10:23.020548 29766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:10:23 dist-test-slave-fjww krb5kdc[29753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565023, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:10:23.030791 29766 init.cc:377] Logged in from keytab as kudu/127.6.192.62@KRBTEST.COM (short username kudu)
I20260928 03:10:23.032644 29766 webserver.cc:460] Webserver started at http://127.6.192.62:34829/ using document root <none> and password file <none>
I20260928 03:10:23.033455 29766 fs_manager.cc:359] Metadata directory not provided
I20260928 03:10:23.033612 29766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:10:23.033980 29766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:10:23.037463 29766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-0/data/instance:
uuid: "f72a9006384545468df49951fa7f2a1b"
format_stamp: "Formatted at 2026-09-28 03:10:23 on dist-test-slave-fjww"
server_key: "939a709f94d1b847859102a8cda9fb27"
server_key_iv: "a0d8b99c15799b799a2913315ef111f6"
server_key_version: "encryptionkey@0"
I20260928 03:10:23.038391 29766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-0/wal/instance:
uuid: "f72a9006384545468df49951fa7f2a1b"
format_stamp: "Formatted at 2026-09-28 03:10:23 on dist-test-slave-fjww"
server_key: "939a709f94d1b847859102a8cda9fb27"
server_key_iv: "a0d8b99c15799b799a2913315ef111f6"
server_key_version: "encryptionkey@0"
I20260928 03:10:23.044580 29766 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 03:10:23.048702 29781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:10:23.050439 29766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 03:10:23.050658 29766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-0/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-0/wal
uuid: "f72a9006384545468df49951fa7f2a1b"
format_stamp: "Formatted at 2026-09-28 03:10:23 on dist-test-slave-fjww"
server_key: "939a709f94d1b847859102a8cda9fb27"
server_key_iv: "a0d8b99c15799b799a2913315ef111f6"
server_key_version: "encryptionkey@0"
I20260928 03:10:23.050922 29766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:10:23.076196 29766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:10:23.081466 29766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:10:23.081791 29766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:10:23.102306 29766 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.62:46039
I20260928 03:10:23.102360 29832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.62:46039 every 8 connection(s)
I20260928 03:10:23.103760 29766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-0/data/info.pb
I20260928 03:10:23.108188 29833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:10:23.110677 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 29766
I20260928 03:10:23.110980 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-0/wal/instance
I20260928 03:10:23.111620 6912 external_mini_cluster.cc:1428] Setting key b9b05ab5befb926dafbb2882e783d10d
I20260928 03:10:23.116210 29833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:10:23 dist-test-slave-fjww krb5kdc[29753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565023, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.192.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.61@KRBTEST.COM" created.
I20260928 03:10:23.145042 29837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.119956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56062 (local address 127.6.192.62:46039)
0928 03:10:23.121829 (+ 1873us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:23.121840 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:23.121899 (+ 59us) server_negotiation.cc:407] Connection header received
0928 03:10:23.124220 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:23.124233 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:23.124661 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:23.124922 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:23.126318 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:23.128727 (+ 2409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:23.130335 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:23.131575 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:23.131812 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:23.136141 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:23.136150 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:23.136165 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:23.136209 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:23.139721 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:23.140656 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:23.140672 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:23.140675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:23.140971 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:23.141550 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:23.141554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:23.141557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:23.142336 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:23.142512 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:23.143263 (+ 751us) server_negotiation.cc:299] Negotiation successful
0928 03:10:23.143641 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":1609,"spinlock_wait_cycles":18176,"thread_start_us":1346,"threads_started":1}
I20260928 03:10:23.145094 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.119956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:46039 (local address 127.0.0.1:56062)
0928 03:10:23.121226 (+ 1270us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.121701 (+ 475us) client_negotiation.cc:174] Beginning negotiation
0928 03:10:23.123750 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:10:23.125038 (+ 1288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:10:23.125044 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:10:23.125686 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:10:23.126065 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:23.126084 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:23.128966 (+ 2882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:23.128970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:23.130150 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:23.130164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:23.131815 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:23.131821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:23.132080 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:23.133009 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:10:23.133049 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:10:23.135814 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:10:23.139918 (+ 4104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:23.139930 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:23.139941 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:23.140487 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:23.141167 (+ 680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:23.141170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:23.141173 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:23.141396 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:23.142514 (+ 1118us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:10:23.142522 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:10:23.142811 (+ 289us) client_negotiation.cc:769] Sending connection context
0928 03:10:23.143121 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 03:10:23.143920 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":206,"thread_start_us":574,"threads_started":1}
I20260928 03:10:23.148322 29833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:10:23.149452 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.149025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:43865 (local address 127.0.0.1:35158)
0928 03:10:23.149199 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.149333 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:43865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 03:10:23.149819 29785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.192.61:43865: connect: Connection refused (error 111)
I20260928 03:10:23.151645 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.151155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:43865 (local address 127.0.0.1:35172)
0928 03:10:23.151325 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.151406 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:43865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 03:10:23.151918 29833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.61:43865: Network error: Client connection negotiation failed: client connection to 127.6.192.61:43865: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.61.keytab.
Entry for principal kudu/127.6.192.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.61.keytab.
I20260928 03:10:23.179052 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.61:43865
--webserver_interface=127.6.192.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.61.keytab
--principal=kudu/127.6.192.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:46039,127.6.192.61:43865,127.6.192.60:40075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:10:23.201321 29833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } attempt: 1
I20260928 03:10:23.202618 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.202129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:43865 (local address 127.0.0.1:35176)
0928 03:10:23.202368 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.202475 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:43865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 03:10:23.203774 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.203354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:43865 (local address 127.0.0.1:35190)
0928 03:10:23.203545 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.203616 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:43865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 03:10:23.203965 29833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.61:43865: Network error: Client connection negotiation failed: client connection to 127.6.192.61:43865: connect: Connection refused (error 111)
I20260928 03:10:23.272297 29833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } attempt: 2
I20260928 03:10:23.273641 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.273127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:43865 (local address 127.0.0.1:35206)
0928 03:10:23.273370 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.273476 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:43865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 03:10:23.274781 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.274366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:43865 (local address 127.0.0.1:35214)
0928 03:10:23.274530 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.274631 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:43865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 03:10:23.274995 29833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.61:43865: Network error: Client connection negotiation failed: client connection to 127.6.192.61:43865: connect: Connection refused (error 111)
I20260928 03:10:23.366253 29833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } attempt: 3
I20260928 03:10:23.367730 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.367138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:43865 (local address 127.0.0.1:35224)
0928 03:10:23.367406 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.367572 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:43865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260928 03:10:23.368938 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.368531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:43865 (local address 127.0.0.1:35232)
0928 03:10:23.368689 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.368785 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:43865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 03:10:23.369240 29833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.61:43865: Network error: Client connection negotiation failed: client connection to 127.6.192.61:43865: connect: Connection refused (error 111)
I20260928 03:10:23.512542 29833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } attempt: 4
I20260928 03:10:23.513964 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.513410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:43865 (local address 127.0.0.1:35246)
0928 03:10:23.513725 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.513815 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:43865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260928 03:10:23.515033 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.514635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:43865 (local address 127.0.0.1:35248)
0928 03:10:23.514786 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.514894 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.61:43865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 03:10:23.515199 29833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.61:43865: Network error: Client connection negotiation failed: client connection to 127.6.192.61:43865: connect: Connection refused (error 111)
W20260928 03:10:23.528578 29840 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 03:10:23.529120 29840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:10:23.529268 29840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:10:23.529426 29840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 03:10:23.537859 29840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:10:23.538005 29840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:10:23.538116 29840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:10:23.538208 29840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:10:23.538319 29840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:10:23.548810 29840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:46039,127.6.192.61:43865,127.6.192.60:40075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.61.keytab
--principal=kudu/127.6.192.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.61:43865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.192.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:10:23.550215 29840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:10:23.551638 29840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:10:23.564344 29845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:23.566287 29848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:23.566313 29846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:10:23.568334 29840 server_base.cc:1034] running on GCE node
I20260928 03:10:23.569235 29840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:10:23.570783 29840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:10:23.572046 29840 hybrid_clock.cc:648] HybridClock initialized: now 1790565023571986 us; error 47 us; skew 500 ppm
I20260928 03:10:23.572428 29840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:10:23 dist-test-slave-fjww krb5kdc[29753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565023, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:10:23.576082 29840 init.cc:377] Logged in from keytab as kudu/127.6.192.61@KRBTEST.COM (short username kudu)
I20260928 03:10:23.578001 29840 webserver.cc:460] Webserver started at http://127.6.192.61:34475/ using document root <none> and password file <none>
I20260928 03:10:23.578774 29840 fs_manager.cc:359] Metadata directory not provided
I20260928 03:10:23.578919 29840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:10:23.579253 29840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:10:23.582422 29840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-1/data/instance:
uuid: "f001c3fcdd97463a8349915f5b672eba"
format_stamp: "Formatted at 2026-09-28 03:10:23 on dist-test-slave-fjww"
server_key: "7e9dbf32a50c34aab086ca3637bdcaf8"
server_key_iv: "8c70eec450c135a6da4a1ab823a91d68"
server_key_version: "encryptionkey@0"
I20260928 03:10:23.583240 29840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-1/wal/instance:
uuid: "f001c3fcdd97463a8349915f5b672eba"
format_stamp: "Formatted at 2026-09-28 03:10:23 on dist-test-slave-fjww"
server_key: "7e9dbf32a50c34aab086ca3637bdcaf8"
server_key_iv: "8c70eec450c135a6da4a1ab823a91d68"
server_key_version: "encryptionkey@0"
I20260928 03:10:23.589079 29840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 03:10:23.593158 29855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:10:23.594985 29840 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 03:10:23.595220 29840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-1/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-1/wal
uuid: "f001c3fcdd97463a8349915f5b672eba"
format_stamp: "Formatted at 2026-09-28 03:10:23 on dist-test-slave-fjww"
server_key: "7e9dbf32a50c34aab086ca3637bdcaf8"
server_key_iv: "8c70eec450c135a6da4a1ab823a91d68"
server_key_version: "encryptionkey@0"
I20260928 03:10:23.595511 29840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:10:23.620460 29840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:10:23.625464 29840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:10:23.625773 29840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:10:23.645646 29840 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.61:43865
I20260928 03:10:23.645668 29906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.61:43865 every 8 connection(s)
I20260928 03:10:23.647221 29840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-1/data/info.pb
I20260928 03:10:23.647733 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 29840
I20260928 03:10:23.647924 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-1/wal/instance
I20260928 03:10:23.648545 6912 external_mini_cluster.cc:1428] Setting key 54b795188f261e809aace01c1d97e0d2
I20260928 03:10:23.652262 29907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:10:23.661839 29907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:10:23 dist-test-slave-fjww krb5kdc[29753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565023, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.61@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.192.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.192.60@KRBTEST.COM" created.
I20260928 03:10:23.685832 29910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.664724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.6.192.62:46039)
0928 03:10:23.665327 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:23.665331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:23.666489 (+ 1158us) server_negotiation.cc:407] Connection header received
0928 03:10:23.668559 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:23.668565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:23.668661 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:23.668780 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:23.670400 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:23.671165 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:23.672937 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:23.673363 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:23.673503 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:23.677953 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:23.677960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:23.677963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:23.678009 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:23.681913 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:23.682898 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:23.682906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:23.682913 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:23.683077 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:23.683662 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:23.683670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:23.683676 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:23.684083 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:23.684279 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:23.685050 (+ 771us) server_negotiation.cc:299] Negotiation successful
0928 03:10:23.685331 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":362,"threads_started":1}
I20260928 03:10:23.686538 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.665408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:46039 (local address 127.0.0.1:56068)
0928 03:10:23.666256 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.666295 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 03:10:23.668266 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:10:23.669052 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:10:23.669061 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:10:23.669613 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:10:23.670177 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:23.670194 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:23.671339 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:23.671343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:23.672783 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:23.672796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:23.673615 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:23.673623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:23.673877 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:23.674649 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:10:23.674673 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:10:23.677583 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:10:23.682140 (+ 4557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:23.682150 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:23.682167 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:23.682725 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:23.683201 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:23.683205 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:23.683207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:23.683524 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:23.684218 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:10:23.684224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:10:23.684637 (+ 413us) client_negotiation.cc:769] Sending connection context
0928 03:10:23.684912 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 03:10:23.685329 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":382,"threads_started":1}
I20260928 03:10:23.688752 29907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:10:23 dist-test-slave-fjww krb5kdc[29753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565023, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.61@KRBTEST.COM for kudu/127.6.192.61@KRBTEST.COM
I20260928 03:10:23.704423 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.689529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:43865 (local address 127.0.0.1:35260)
0928 03:10:23.689709 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.689734 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 03:10:23.689826 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:10:23.691032 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:10:23.691036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:10:23.691585 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:10:23.691747 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:23.691756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:23.693113 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:23.693118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:23.693967 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:23.693978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:23.694837 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:23.694843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:23.695041 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:23.695725 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:10:23.695742 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:10:23.697855 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:10:23.701826 (+ 3971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:23.701832 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:23.701836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:23.702215 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:23.702594 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:23.702600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:23.702603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:23.702747 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:23.703350 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:10:23.703355 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:10:23.703536 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 03:10:23.703682 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 03:10:23.704017 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":1}
I20260928 03:10:23.704530 29913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.689898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.6.192.61:43865)
0928 03:10:23.690554 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:23.690560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:23.690586 (+ 26us) server_negotiation.cc:407] Connection header received
0928 03:10:23.690636 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:23.690641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:23.690829 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:23.691045 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:23.691930 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:23.692973 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:23.694117 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:23.694670 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:23.694824 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:23.698078 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:23.698091 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:23.698105 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:23.698139 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:23.701657 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:23.702353 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:23.702368 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:23.702370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:23.702486 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:23.702848 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:23.702855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:23.702859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:23.703219 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:23.703439 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:23.703827 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 03:10:23.704137 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":506,"thread_start_us":362,"threads_started":1}
I20260928 03:10:23.707026 29907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:10:23.708412 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.707792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40496)
0928 03:10:23.708082 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.708230 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260928 03:10:23.708683 29857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
I20260928 03:10:23.710531 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.710084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40510)
0928 03:10:23.710277 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.710378 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 03:10:23.710796 29907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:40075: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.192.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.60.keytab.
Entry for principal kudu/127.6.192.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.60.keytab.
I20260928 03:10:23.716610 6912 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97hosG/build/asan/bin/kudu
/tmp/dist-test-task97hosG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.60:40075
--webserver_interface=127.6.192.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.60.keytab
--principal=kudu/127.6.192.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.192.20:41367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.192.62:46039,127.6.192.61:43865,127.6.192.60:40075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 03:10:23.760108 29907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } attempt: 1
I20260928 03:10:23.761373 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.760893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40516)
0928 03:10:23.761094 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.761201 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 03:10:23.762519 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.762133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40522)
0928 03:10:23.762332 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.762410 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 03:10:23.762745 29907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:40075: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
I20260928 03:10:23.814476 29833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } attempt: 5
Sep 28 03:10:23 dist-test-slave-fjww krb5kdc[29753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565023, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for kudu/127.6.192.61@KRBTEST.COM
I20260928 03:10:23.829385 29913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.815477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35272 (local address 127.6.192.61:43865)
0928 03:10:23.815677 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:23.815682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:23.815706 (+ 24us) server_negotiation.cc:407] Connection header received
0928 03:10:23.815799 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:23.815804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:23.815886 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:23.816005 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:23.816891 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:23.817587 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:23.818801 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:23.819251 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:23.819395 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:23.823151 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:23.823157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:23.823160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:23.823219 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:23.826235 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:23.827226 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:23.827232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:23.827234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:23.827398 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:23.828011 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:23.828015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:23.828017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:23.828329 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:23.828451 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:23.828959 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 03:10:23.829139 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 03:10:23.829476 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.815337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:43865 (local address 127.0.0.1:35272)
0928 03:10:23.815574 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.815602 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 03:10:23.815707 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:10:23.816046 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:10:23.816049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:10:23.816468 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:10:23.816643 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:23.816652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:23.817746 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:23.817753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:23.818637 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:23.818649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:23.819411 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:23.819417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:23.819677 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:23.820394 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:10:23.820413 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:10:23.822873 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:10:23.826420 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:23.826427 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:23.826431 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:23.827031 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:23.827598 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:23.827604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:23.827607 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:23.827795 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:23.828454 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:10:23.828459 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:10:23.828634 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 03:10:23.828829 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 03:10:23.829129 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 03:10:23.830698 29833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:10:23.830968 29907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } attempt: 2
I20260928 03:10:23.832036 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.831544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40540)
0928 03:10:23.831773 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.831872 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 03:10:23.832162 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.831643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40532)
0928 03:10:23.831899 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.831975 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260928 03:10:23.833202 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.832763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40546)
0928 03:10:23.832945 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.833050 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 03:10:23.833370 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.832932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40558)
0928 03:10:23.833108 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.833222 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 03:10:23.833422 29907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:40075: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
W20260928 03:10:23.833693 29833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:40075: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
I20260928 03:10:23.884980 29833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } attempt: 1
I20260928 03:10:23.886366 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.885807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40570)
0928 03:10:23.886040 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.886149 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 03:10:23.887605 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.887138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40572)
0928 03:10:23.887303 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.887400 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 03:10:23.887828 29833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:40075: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
I20260928 03:10:23.924710 29907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } attempt: 3
I20260928 03:10:23.926090 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.925515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40588)
0928 03:10:23.925744 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.925875 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 03:10:23.927412 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.926918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40596)
0928 03:10:23.927192 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.927277 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260928 03:10:23.927752 29907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:40075: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
I20260928 03:10:23.956104 29833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } attempt: 2
I20260928 03:10:23.957319 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.956807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40598)
0928 03:10:23.957018 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.957142 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 03:10:23.958698 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:23.958168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40612)
0928 03:10:23.958465 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:23.958547 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260928 03:10:23.958977 29833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:40075: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
I20260928 03:10:24.065237 29833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } attempt: 3
I20260928 03:10:24.066498 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.065968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40620)
0928 03:10:24.066209 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:24.066310 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 03:10:24.067756 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.067263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40622)
0928 03:10:24.067528 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:24.067604 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260928 03:10:24.067657 29915 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 03:10:24.067996 29833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:40075: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
W20260928 03:10:24.068221 29915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 03:10:24.068356 29915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 03:10:24.068516 29915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 03:10:24.071009 29907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } attempt: 4
I20260928 03:10:24.072250 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.071701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40626)
0928 03:10:24.072013 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:24.072093 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260928 03:10:24.073294 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.072962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40628)
0928 03:10:24.073125 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:24.073190 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 03:10:24.073519 29907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.192.60:40075: Network error: Client connection negotiation failed: client connection to 127.6.192.60:40075: connect: Connection refused (error 111)
W20260928 03:10:24.077737 29915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 03:10:24.077852 29915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 03:10:24.077956 29915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 03:10:24.078033 29915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 03:10:24.078118 29915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 03:10:24.090241 29915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.192.20:41367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.192.62:46039,127.6.192.61:43865,127.6.192.60:40075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/krb5kdc/kudu_127.6.192.60.keytab
--principal=kudu/127.6.192.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.192.60:40075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.192.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 03:10:24.091864 29915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 03:10:24.093434 29915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 03:10:24.107349 29921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:24.107465 29920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 03:10:24.107414 29923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 03:10:24.109557 29915 server_base.cc:1034] running on GCE node
I20260928 03:10:24.110478 29915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 03:10:24.112097 29915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 03:10:24.113348 29915 hybrid_clock.cc:648] HybridClock initialized: now 1790565024113244 us; error 84 us; skew 500 ppm
I20260928 03:10:24.113714 29915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 03:10:24 dist-test-slave-fjww krb5kdc[29753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565024, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 03:10:24.117130 29915 init.cc:377] Logged in from keytab as kudu/127.6.192.60@KRBTEST.COM (short username kudu)
I20260928 03:10:24.118705 29915 webserver.cc:460] Webserver started at http://127.6.192.60:37525/ using document root <none> and password file <none>
I20260928 03:10:24.119486 29915 fs_manager.cc:359] Metadata directory not provided
I20260928 03:10:24.119617 29915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 03:10:24.119902 29915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 03:10:24.122958 29915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-2/data/instance:
uuid: "bb30c63cba33444eba313d1530dc1b6e"
format_stamp: "Formatted at 2026-09-28 03:10:24 on dist-test-slave-fjww"
server_key: "cc225df06938b84725094f2e7959b580"
server_key_iv: "2b6280d347ab87f017f0ac1774391dd0"
server_key_version: "encryptionkey@0"
I20260928 03:10:24.123788 29915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-2/wal/instance:
uuid: "bb30c63cba33444eba313d1530dc1b6e"
format_stamp: "Formatted at 2026-09-28 03:10:24 on dist-test-slave-fjww"
server_key: "cc225df06938b84725094f2e7959b580"
server_key_iv: "2b6280d347ab87f017f0ac1774391dd0"
server_key_version: "encryptionkey@0"
I20260928 03:10:24.129487 29915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 03:10:24.133392 29930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 03:10:24.135156 29915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 03:10:24.135371 29915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-2/data,/tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-2/wal
uuid: "bb30c63cba33444eba313d1530dc1b6e"
format_stamp: "Formatted at 2026-09-28 03:10:24 on dist-test-slave-fjww"
server_key: "cc225df06938b84725094f2e7959b580"
server_key_iv: "2b6280d347ab87f017f0ac1774391dd0"
server_key_version: "encryptionkey@0"
I20260928 03:10:24.135689 29915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 03:10:24.163002 29915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 03:10:24.167650 29915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 03:10:24.167996 29915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 03:10:24.189107 29915 rpc_server.cc:307] RPC server started. Bound to: 127.6.192.60:40075
I20260928 03:10:24.189113 29981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.192.60:40075 every 8 connection(s)
I20260928 03:10:24.190652 29915 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-2/data/info.pb
I20260928 03:10:24.194857 29982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 03:10:24.194972 6912 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97hosG/build/asan/bin/kudu as pid 29915
I20260928 03:10:24.195180 6912 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97hosG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790564644636890-6912-0/minicluster-data/master-2/wal/instance
I20260928 03:10:24.195751 6912 external_mini_cluster.cc:1428] Setting key e60877da4312926d0f23650453739faa
I20260928 03:10:24.204778 29982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:10:24 dist-test-slave-fjww krb5kdc[29753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.60@KRBTEST.COM
Sep 28 03:10:24 dist-test-slave-fjww krb5kdc[29753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565024, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.60@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:10:24.223373 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.197671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40638 (local address 127.6.192.60:40075)
0928 03:10:24.198613 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.198626 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.198701 (+ 75us) server_negotiation.cc:407] Connection header received
0928 03:10:24.201520 (+ 2819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.201549 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.202081 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.202491 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.204340 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.205984 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.207742 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.208922 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.209103 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:24.214030 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:24.214043 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:24.214061 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:24.214124 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:24.218328 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.219085 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.219098 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.219111 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.219316 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.219772 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.219786 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.219792 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.220621 (+ 829us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:24.221017 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:24.221363 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 03:10:24.221883 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":431,"threads_started":1}
I20260928 03:10:24.225137 29986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.207665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.62:46039 (local address 127.0.0.1:56074)
0928 03:10:24.208446 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:24.208491 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 03:10:24.208648 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:10:24.209536 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:10:24.209547 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:10:24.210494 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:10:24.210849 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:24.210864 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.211947 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.211954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:24.213357 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:24.213374 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.214522 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.214526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:24.214675 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:24.215562 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:10:24.215583 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:10:24.218209 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:10:24.222273 (+ 4064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:24.222282 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:24.222287 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:24.222742 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:24.223162 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:24.223165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:24.223167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:24.223302 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:24.224005 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:10:24.224012 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:10:24.224210 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 03:10:24.224438 (+ 228us) client_negotiation.cc:240] Negotiation successful
0928 03:10:24.224746 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":309,"threads_started":1}
I20260928 03:10:24.225140 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.207680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.6.192.62:46039)
0928 03:10:24.209116 (+ 1436us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.209122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.209140 (+ 18us) server_negotiation.cc:407] Connection header received
0928 03:10:24.209199 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.209204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.209301 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.209417 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.211029 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.211779 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.213560 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.214249 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.214470 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:24.218455 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:24.218464 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:24.218470 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:24.218514 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:24.222113 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.222871 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.222877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.222880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.223047 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.223486 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.223493 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.223499 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.223864 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:24.223997 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:24.224548 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 03:10:24.224719 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1308,"thread_start_us":1175,"threads_started":1}
I20260928 03:10:24.227111 29982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 03:10:24 dist-test-slave-fjww krb5kdc[29753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565024, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.60@KRBTEST.COM for kudu/127.6.192.61@KRBTEST.COM
Sep 28 03:10:24 dist-test-slave-fjww krb5kdc[29753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.62@KRBTEST.COM
I20260928 03:10:24.242565 29986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.227798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.61:43865 (local address 127.0.0.1:35274)
0928 03:10:24.228129 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:24.228147 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 03:10:24.228261 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:10:24.228605 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:10:24.228612 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:10:24.229062 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:10:24.229248 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:24.229261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.230285 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.230293 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:24.231172 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:24.231182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.232032 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.232038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:24.232220 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:24.232847 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:10:24.232873 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:10:24.235552 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:10:24.239537 (+ 3985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:24.239544 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:24.239549 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:24.240028 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:24.240547 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:24.240553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:24.240558 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:24.240730 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:24.241495 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:10:24.241501 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:10:24.241699 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 03:10:24.241840 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 03:10:24.242139 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260928 03:10:24.242741 29913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.227918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35274 (local address 127.6.192.61:43865)
0928 03:10:24.228100 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.228108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.228213 (+ 105us) server_negotiation.cc:407] Connection header received
0928 03:10:24.228393 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.228397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.228459 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.228541 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.229456 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.230149 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.231326 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.231878 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.232019 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:24.235793 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:24.235800 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:24.235805 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:24.235849 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:24.239192 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.240180 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.240186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.240191 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.240383 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.240955 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.240964 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.240969 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.241355 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:24.241491 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:24.242018 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 03:10:24.242243 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 03:10:24.243955 29982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } has no permanent_uuid. Determining permanent_uuid...
I20260928 03:10:24.244774 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.227153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.6.192.62:46039)
0928 03:10:24.227400 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.227406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.227426 (+ 20us) server_negotiation.cc:407] Connection header received
0928 03:10:24.227531 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.227537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.227638 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.227761 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.229361 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.230125 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.231521 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.231962 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.232108 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:24.237335 (+ 5227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:24.237341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:24.237344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:24.237382 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:24.240867 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.241678 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.241683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.241686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.241909 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.242429 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.242436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.242441 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.242865 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:24.243186 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:24.244044 (+ 858us) server_negotiation.cc:299] Negotiation successful
0928 03:10:24.244322 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 03:10:24.245268 29833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } attempt: 4
Sep 28 03:10:24 dist-test-slave-fjww krb5kdc[29753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565024, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.60@KRBTEST.COM for kudu/127.6.192.60@KRBTEST.COM
Sep 28 03:10:24 dist-test-slave-fjww krb5kdc[29753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565023, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.62@KRBTEST.COM for kudu/127.6.192.60@KRBTEST.COM
Sep 28 03:10:24 dist-test-slave-fjww krb5kdc[29753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.192.61@KRBTEST.COM
I20260928 03:10:24.263401 29986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.244840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40640)
0928 03:10:24.245338 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:24.245363 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 03:10:24.245502 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:10:24.246048 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:10:24.246053 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:10:24.246709 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:10:24.246911 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:24.246920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.248623 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.248629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:24.250260 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:24.250275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.251384 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.251391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:24.251749 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:24.252478 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:10:24.252504 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:10:24.255516 (+ 3012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:10:24.259968 (+ 4452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:24.259977 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:24.259984 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:24.260652 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:24.261116 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:24.261125 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:24.261133 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:24.261349 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:24.262262 (+ 913us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:10:24.262272 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:10:24.262471 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 03:10:24.262743 (+ 272us) client_negotiation.cc:240] Negotiation successful
0928 03:10:24.263068 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":3}
I20260928 03:10:24.264962 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.245100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40640 (local address 127.6.192.60:40075)
0928 03:10:24.245274 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.245346 (+ 72us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.245463 (+ 117us) server_negotiation.cc:407] Connection header received
0928 03:10:24.245629 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.245652 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.245744 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.245860 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.247051 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.248450 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.250432 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.251196 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.251367 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:24.255732 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:24.255742 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:24.255751 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:24.255798 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:24.259721 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.260800 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.260808 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.260814 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.260971 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.261530 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.261536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.261541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.261999 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:24.262183 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:24.264199 (+ 2016us) server_negotiation.cc:299] Negotiation successful
0928 03:10:24.264465 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":54}
I20260928 03:10:24.266546 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.246137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.6.192.60:40075)
0928 03:10:24.247725 (+ 1588us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.247731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.247760 (+ 29us) server_negotiation.cc:407] Connection header received
0928 03:10:24.247820 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.247825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.247891 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.248089 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.249033 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.250479 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.252334 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.253084 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.253203 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:24.257668 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:24.257683 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:24.257700 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:24.257761 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:24.262567 (+ 4806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.263735 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.263740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.263743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.263925 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.264355 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.264361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.264366 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.264657 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:24.264955 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:24.265819 (+ 864us) server_negotiation.cc:299] Negotiation successful
0928 03:10:24.266044 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":1445,"thread_start_us":1145,"threads_started":1}
I20260928 03:10:24.266826 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.246042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40648)
0928 03:10:24.246318 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:24.246339 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 03:10:24.246446 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:10:24.248119 (+ 1673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:10:24.248127 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:10:24.248623 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:10:24.248824 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:24.248837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.250617 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.250623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:24.252139 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:24.252163 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.253272 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.253280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:24.253542 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:24.254402 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:10:24.254434 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:10:24.257344 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:10:24.262756 (+ 5412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:24.262763 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:24.262767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:24.263410 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:24.264069 (+ 659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:24.264073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:24.264076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:24.264215 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:24.264790 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:10:24.264795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:10:24.265030 (+ 235us) client_negotiation.cc:769] Sending connection context
0928 03:10:24.265652 (+ 622us) client_negotiation.cc:240] Negotiation successful
0928 03:10:24.265990 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260928 03:10:24.268091 29913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.247626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.6.192.61:43865)
0928 03:10:24.247851 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.247857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.247880 (+ 23us) server_negotiation.cc:407] Connection header received
0928 03:10:24.247993 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.248000 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.248080 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.249782 (+ 1702us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.249918 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.250970 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.253000 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.253592 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.253735 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:24.259998 (+ 6263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:24.260005 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:24.260011 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:24.260061 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:24.263743 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.264790 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.264798 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.264802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.265000 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.265506 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.265512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.265516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.265890 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:24.266187 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:24.267355 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0928 03:10:24.267671 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 03:10:24.272437 29982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e: Bootstrap starting.
I20260928 03:10:24.273105 6912 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 03:10:24.274076 29833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b: Bootstrap starting.
I20260928 03:10:24.277416 29982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e: Neither blocks nor log segments found. Creating new log.
I20260928 03:10:24.279156 29982 log.cc:824] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e: Log is configured to *not* fsync() on all Append() calls
I20260928 03:10:24.275909 29808 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f72a9006384545468df49951fa7f2a1b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56088
I20260928 03:10:24.279356 29833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b: Neither blocks nor log segments found. Creating new log.
I20260928 03:10:24.280877 29833 log.cc:824] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b: Log is configured to *not* fsync() on all Append() calls
I20260928 03:10:24.283631 29982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e: No bootstrap required, opened a new log
I20260928 03:10:24.286114 29833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b: No bootstrap required, opened a new log
I20260928 03:10:24.295598 29833 raft_consensus.cc:348] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72a9006384545468df49951fa7f2a1b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } } peers { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } } peers { permanent_uuid: "bb30c63cba33444eba313d1530dc1b6e" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } }
I20260928 03:10:24.296169 29833 raft_consensus.cc:729] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f72a9006384545468df49951fa7f2a1b, State: Initialized, Role: FOLLOWER
I20260928 03:10:24.296833 29833 consensus_queue.cc:260] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72a9006384545468df49951fa7f2a1b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } } peers { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } } peers { permanent_uuid: "bb30c63cba33444eba313d1530dc1b6e" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } }
I20260928 03:10:24.296943 29982 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72a9006384545468df49951fa7f2a1b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } } peers { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } } peers { permanent_uuid: "bb30c63cba33444eba313d1530dc1b6e" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } }
I20260928 03:10:24.297446 29982 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb30c63cba33444eba313d1530dc1b6e, State: Initialized, Role: FOLLOWER
I20260928 03:10:24.298067 29982 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72a9006384545468df49951fa7f2a1b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } } peers { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } } peers { permanent_uuid: "bb30c63cba33444eba313d1530dc1b6e" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } }
I20260928 03:10:24.298074 29991 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72a9006384545468df49951fa7f2a1b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } } peers { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } } peers { permanent_uuid: "bb30c63cba33444eba313d1530dc1b6e" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } } }
I20260928 03:10:24.298545 29991 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:10:24.299247 29992 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72a9006384545468df49951fa7f2a1b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } } peers { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } } peers { permanent_uuid: "bb30c63cba33444eba313d1530dc1b6e" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } } }
I20260928 03:10:24.299847 29992 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:10:24.300045 29833 sys_catalog.cc:564] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:10:24.300601 29808 raft_consensus.cc:482] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 03:10:24.300812 29808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:10:24.300838 29982 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:10:24.304415 29808 raft_consensus.cc:504] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72a9006384545468df49951fa7f2a1b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } } peers { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } } peers { permanent_uuid: "bb30c63cba33444eba313d1530dc1b6e" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } }
I20260928 03:10:24.306166 29808 leader_election.cc:288] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [CANDIDATE]: Term 1 election: Requested vote from peers f001c3fcdd97463a8349915f5b672eba (127.6.192.61:43865), bb30c63cba33444eba313d1530dc1b6e (127.6.192.60:40075)
I20260928 03:10:24.307549 29957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f72a9006384545468df49951fa7f2a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bb30c63cba33444eba313d1530dc1b6e"
I20260928 03:10:24.308452 29957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:10:24.314311 29957 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f72a9006384545468df49951fa7f2a1b in term 1.
I20260928 03:10:24.315418 29783 leader_election.cc:302] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb30c63cba33444eba313d1530dc1b6e, f72a9006384545468df49951fa7f2a1b; no voters:
I20260928 03:10:24.316370 29991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 03:10:24.325042 29991 raft_consensus.cc:686] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [term 1 LEADER]: Becoming Leader. State: Replica: f72a9006384545468df49951fa7f2a1b, State: Running, Role: LEADER
I20260928 03:10:24.326749 29991 consensus_queue.cc:237] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72a9006384545468df49951fa7f2a1b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } } peers { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } } peers { permanent_uuid: "bb30c63cba33444eba313d1530dc1b6e" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } }
W20260928 03:10:24.334980 29787 tablet.cc:2369] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:10:24.334774 30009 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f72a9006384545468df49951fa7f2a1b. Latest consensus state: current_term: 1 leader_uuid: "f72a9006384545468df49951fa7f2a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72a9006384545468df49951fa7f2a1b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } } peers { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } } peers { permanent_uuid: "bb30c63cba33444eba313d1530dc1b6e" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } } }
I20260928 03:10:24.335363 30009 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:24.338630 29766 master_runner.cc:186] Error getting master registration for 127.6.192.61:43865: OK, instance_id { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" instance_seqno: 1790565023640065 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 03:10:24.351364 29766 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 03:10:24.368383 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.329003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40650 (local address 127.6.192.60:40075)
0928 03:10:24.329330 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.329340 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.330462 (+ 1122us) server_negotiation.cc:407] Connection header received
0928 03:10:24.332152 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.332158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.332260 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.332370 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.341705 (+ 9335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.343583 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.347206 (+ 3623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.347970 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.348077 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:24.354989 (+ 6912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:24.355000 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:24.355006 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:24.355075 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:24.360356 (+ 5281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.362618 (+ 2262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.362630 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.362637 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.363046 (+ 409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.364316 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.364354 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.364360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.365277 (+ 917us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:24.365859 (+ 582us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:24.367316 (+ 1457us) server_negotiation.cc:299] Negotiation successful
0928 03:10:24.367758 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 03:10:24.370843 30020 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 03:10:24.371114 30020 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 03:10:24.371321 30021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 03:10:24.372735 29907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } attempt: 5
I20260928 03:10:24.373689 30021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 03:10:24.389233 29957 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e [term 1 FOLLOWER]: Refusing update from remote peer f72a9006384545468df49951fa7f2a1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:10:24.390302 29913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.332494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.6.192.61:43865)
0928 03:10:24.332663 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.332670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.333542 (+ 872us) server_negotiation.cc:407] Connection header received
0928 03:10:24.333697 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.333702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.333780 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.333888 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.341375 (+ 7487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.343140 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.349969 (+ 6829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.351240 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.351419 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:24.375724 (+ 24305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:24.375737 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:24.375746 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:24.375822 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:24.381685 (+ 5863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.383122 (+ 1437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.383132 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.383137 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.383577 (+ 440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.386681 (+ 3104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.386688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.386692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.387254 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:24.387591 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:24.389451 (+ 1860us) server_negotiation.cc:299] Negotiation successful
0928 03:10:24.389785 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 03:10:24.391492 30009 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb30c63cba33444eba313d1530dc1b6e" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 28 03:10:24 dist-test-slave-fjww krb5kdc[29753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790565023, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.192.61@KRBTEST.COM for kudu/127.6.192.60@KRBTEST.COM
I20260928 03:10:24.404405 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.330573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56100 (local address 127.6.192.62:46039)
0928 03:10:24.330751 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.330757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.332521 (+ 1764us) server_negotiation.cc:407] Connection header received
0928 03:10:24.336407 (+ 3886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.336417 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.336527 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.336622 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.353297 (+ 16675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.355179 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.364417 (+ 9238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.365413 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.365595 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:24.391368 (+ 25773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:24.391377 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:24.391384 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:24.391486 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:24.396086 (+ 4600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.397371 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.397380 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.397384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.397667 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.398397 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.398405 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.398408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.399133 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:24.400241 (+ 1108us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:24.401272 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0928 03:10:24.402162 (+ 890us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 03:10:24.407366 30025 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 03:10:24.410092 30025 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 03:10:24.414734 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.373363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.192.60:40075 (local address 127.0.0.1:40654)
0928 03:10:24.373754 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0928 03:10:24.373777 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 03:10:24.373895 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 03:10:24.374982 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 03:10:24.374986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 03:10:24.375658 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0928 03:10:24.375903 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:24.375920 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.378588 (+ 2668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.378603 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:24.380810 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 03:10:24.380839 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.387658 (+ 6819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.387666 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 03:10:24.387918 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:24.393604 (+ 5686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 03:10:24.393641 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 03:10:24.397113 (+ 3472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 03:10:24.411238 (+ 14125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:24.411246 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:24.411252 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:24.411973 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:24.412555 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 03:10:24.412573 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 03:10:24.412578 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 03:10:24.412778 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 03:10:24.413587 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 03:10:24.413594 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 03:10:24.413787 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 03:10:24.413953 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 03:10:24.414292 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250}
I20260928 03:10:24.416229 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:24.374125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40654 (local address 127.6.192.60:40075)
0928 03:10:24.374426 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.374433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.374481 (+ 48us) server_negotiation.cc:407] Connection header received
0928 03:10:24.374573 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.374578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.374744 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.376113 (+ 1369us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.376394 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.378280 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.381246 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:24.382493 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:24.391959 (+ 9466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:24.397490 (+ 5531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 03:10:24.397499 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 03:10:24.397505 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 03:10:24.397564 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 03:10:24.410990 (+ 13426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.412162 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.412194 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.412197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.412426 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 03:10:24.412916 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 03:10:24.412921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 03:10:24.412925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 03:10:24.413383 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 03:10:24.413619 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:24.415541 (+ 1922us) server_negotiation.cc:299] Negotiation successful
0928 03:10:24.415822 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 03:10:24.389997 29915 master_runner.cc:186] Error getting master registration for 127.6.192.61:43865: OK, instance_id { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" instance_seqno: 1790565023640065 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260928 03:10:24.420334 29936 tablet.cc:2369] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 03:10:24.420500 29915 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 03:10:24.421167 29992 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f72a9006384545468df49951fa7f2a1b. Latest consensus state: current_term: 1 leader_uuid: "f72a9006384545468df49951fa7f2a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72a9006384545468df49951fa7f2a1b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } } peers { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } } peers { permanent_uuid: "bb30c63cba33444eba313d1530dc1b6e" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } } }
I20260928 03:10:24.421648 29992 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:10:24.426100 30028 mvcc.cc:205] Tried to move back new op lower bound from 7334154339886985216 to 7334154339654107136. Current Snapshot: MvccSnapshot[applied={T|T < 7334154339886985216}]
I20260928 03:10:24.427047 29992 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f72a9006384545468df49951fa7f2a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72a9006384545468df49951fa7f2a1b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } } peers { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } } peers { permanent_uuid: "bb30c63cba33444eba313d1530dc1b6e" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } } }
I20260928 03:10:24.427354 29992 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:10:24.434548 29907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba: Bootstrap starting.
I20260928 03:10:24.435808 30026 mvcc.cc:205] Tried to move back new op lower bound from 7334154339886985216 to 7334154339654107136. Current Snapshot: MvccSnapshot[applied={T|T < 7334154339886985216}]
I20260928 03:10:24.444327 29907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba: Neither blocks nor log segments found. Creating new log.
I20260928 03:10:24.445992 29907 log.cc:824] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba: Log is configured to *not* fsync() on all Append() calls
I20260928 03:10:24.450686 30021 catalog_manager.cc:1347] Generated new cluster ID: 2c04cf56b8484e7e8661700067c806b3
I20260928 03:10:24.450847 30021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 03:10:24.451431 29991 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f72a9006384545468df49951fa7f2a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72a9006384545468df49951fa7f2a1b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } } peers { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } } peers { permanent_uuid: "bb30c63cba33444eba313d1530dc1b6e" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } } }
I20260928 03:10:24.451855 29991 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [sys.catalog]: This master's current role is: LEADER
I20260928 03:10:24.470424 29907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba: No bootstrap required, opened a new log
I20260928 03:10:24.492962 30021 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 03:10:24.495266 30021 catalog_manager.cc:1504] Loading token signing keys...
I20260928 03:10:24.508394 29907 raft_consensus.cc:348] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72a9006384545468df49951fa7f2a1b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } } peers { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } } peers { permanent_uuid: "bb30c63cba33444eba313d1530dc1b6e" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } }
I20260928 03:10:24.514884 30021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b: Generated new TSK 0
I20260928 03:10:24.516239 30021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 03:10:24.509414 29907 raft_consensus.cc:729] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f001c3fcdd97463a8349915f5b672eba, State: Initialized, Role: FOLLOWER
I20260928 03:10:24.518569 29907 consensus_queue.cc:260] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72a9006384545468df49951fa7f2a1b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } } peers { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } } peers { permanent_uuid: "bb30c63cba33444eba313d1530dc1b6e" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } }
I20260928 03:10:24.520614 30033 sys_catalog.cc:455] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72a9006384545468df49951fa7f2a1b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } } peers { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } } peers { permanent_uuid: "bb30c63cba33444eba313d1530dc1b6e" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } } }
I20260928 03:10:24.522084 29907 sys_catalog.cc:564] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 03:10:24.522145 29882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f72a9006384545468df49951fa7f2a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f001c3fcdd97463a8349915f5b672eba"
I20260928 03:10:24.522688 29882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba [term 0 FOLLOWER]: Advancing to term 1
I20260928 03:10:24.542023 29882 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f72a9006384545468df49951fa7f2a1b in term 1.
I20260928 03:10:24.543365 29881 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba [term 1 FOLLOWER]: Refusing update from remote peer f72a9006384545468df49951fa7f2a1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 03:10:24.546099 30009 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 03:10:24.521875 30033 sys_catalog.cc:458] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:10:24.596491 30039 mvcc.cc:205] Tried to move back new op lower bound from 7334154339886985216 to 7334154339654107136. Current Snapshot: MvccSnapshot[applied={T|T < 7334154339886985216}]
I20260928 03:10:24.607134 30033 sys_catalog.cc:455] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba [sys.catalog]: SysCatalogTable state changed. Reason: New leader f72a9006384545468df49951fa7f2a1b. Latest consensus state: current_term: 1 leader_uuid: "f72a9006384545468df49951fa7f2a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72a9006384545468df49951fa7f2a1b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } } peers { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } } peers { permanent_uuid: "bb30c63cba33444eba313d1530dc1b6e" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } } }
I20260928 03:10:24.607091 30036 sys_catalog.cc:455] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f72a9006384545468df49951fa7f2a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72a9006384545468df49951fa7f2a1b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } } peers { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } } peers { permanent_uuid: "bb30c63cba33444eba313d1530dc1b6e" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } } }
I20260928 03:10:24.607681 30033 sys_catalog.cc:458] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:10:24.607684 30036 sys_catalog.cc:458] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba [sys.catalog]: This master's current role is: FOLLOWER
I20260928 03:10:24.627833 30052 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260928 03:10:24.641494 6912 mini_kdc.cc:348] Time spent kdestroy: real 0.106s user 0.004s sys 0.027s
W20260928 03:10:24.668886 29987 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:24.645066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56116 (local address 127.6.192.62:46039)
0928 03:10:24.645323 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.645329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.652508 (+ 7179us) server_negotiation.cc:407] Connection header received
0928 03:10:24.654270 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.654277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.654386 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.654484 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.668581 (+ 14097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56116: BlockingRecv error: recv got EOF from 127.0.0.1:56116 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260928 03:10:24.671377 29913 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:24.653244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.6.192.61:43865)
0928 03:10:24.653492 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.653499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.667113 (+ 13614us) server_negotiation.cc:407] Connection header received
0928 03:10:24.669717 (+ 2604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.669726 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.669835 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.669910 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.671128 (+ 1218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35288: BlockingRecv error: recv got EOF from 127.0.0.1:35288 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260928 03:10:24.674636 29988 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:24.655582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.6.192.60:40075)
0928 03:10:24.655790 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.655797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.672446 (+ 16649us) server_negotiation.cc:407] Connection header received
0928 03:10:24.673015 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.673025 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.673132 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.673224 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.674323 (+ 1099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40662: BlockingRecv error: recv got EOF from 127.0.0.1:40662 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 03:10:24.700917 29913 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:24.698179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.6.192.61:43865)
0928 03:10:24.698414 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.698423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.699029 (+ 606us) server_negotiation.cc:407] Connection header received
0928 03:10:24.699343 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.699351 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.699587 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.699720 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.700612 (+ 892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35294: BlockingRecv error: recv got EOF from 127.0.0.1:35294 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 03:10:24.708564 29913 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:24.706230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.6.192.61:43865)
0928 03:10:24.706467 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.706476 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.707044 (+ 568us) server_negotiation.cc:407] Connection header received
0928 03:10:24.707288 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.707294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.707390 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.707566 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.708347 (+ 781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35302: BlockingRecv error: recv got EOF from 127.0.0.1:35302 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 03:10:24.716840 29913 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:24.713865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35318 (local address 127.6.192.61:43865)
0928 03:10:24.714741 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.714750 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.715317 (+ 567us) server_negotiation.cc:407] Connection header received
0928 03:10:24.715565 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.715571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.715669 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.715783 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.716552 (+ 769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35318: BlockingRecv error: recv got EOF from 127.0.0.1:35318 (error 108)
Metrics: {"server-negotiator.queue_time_us":689}
W20260928 03:10:24.730700 29913 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:24.728285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35328 (local address 127.6.192.61:43865)
0928 03:10:24.728589 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.728598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.729185 (+ 587us) server_negotiation.cc:407] Connection header received
0928 03:10:24.729428 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.729433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.729524 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.729645 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.730455 (+ 810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35328: BlockingRecv error: recv got EOF from 127.0.0.1:35328 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
W20260928 03:10:24.744917 29913 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:24.742571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35334 (local address 127.6.192.61:43865)
0928 03:10:24.742826 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.742836 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.743422 (+ 586us) server_negotiation.cc:407] Connection header received
0928 03:10:24.743698 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.743704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.743796 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.743913 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.744663 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35334: BlockingRecv error: recv got EOF from 127.0.0.1:35334 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 03:10:24.767198 29913 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:24.764619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35338 (local address 127.6.192.61:43865)
0928 03:10:24.764866 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.764874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.765512 (+ 638us) server_negotiation.cc:407] Connection header received
0928 03:10:24.765841 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.765845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.765994 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.766117 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.766921 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35338: BlockingRecv error: recv got EOF from 127.0.0.1:35338 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 03:10:24.809176 29913 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:24.806545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.6.192.61:43865)
0928 03:10:24.806783 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.806793 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.807600 (+ 807us) server_negotiation.cc:407] Connection header received
0928 03:10:24.807858 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.807864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.807969 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.808088 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.808883 (+ 795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35342: BlockingRecv error: recv got EOF from 127.0.0.1:35342 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 03:10:24.879412 29913 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:24.877109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.6.192.61:43865)
0928 03:10:24.877372 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:24.877381 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:24.877931 (+ 550us) server_negotiation.cc:407] Connection header received
0928 03:10:24.878168 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:24.878178 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:24.878258 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:24.878386 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:24.879180 (+ 794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35358: BlockingRecv error: recv got EOF from 127.0.0.1:35358 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260928 03:10:25.016820 29913 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:25.014112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.6.192.61:43865)
0928 03:10:25.014352 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:25.014358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:25.015103 (+ 745us) server_negotiation.cc:407] Connection header received
0928 03:10:25.015371 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:25.015377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:25.015504 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:25.015626 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:25.016550 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35374: BlockingRecv error: recv got EOF from 127.0.0.1:35374 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 03:10:25.024031 30105 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f72a9006384545468df49951fa7f2a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f72a9006384545468df49951fa7f2a1b" member_type: VOTER last_known_addr { host: "127.6.192.62" port: 46039 } } peers { permanent_uuid: "f001c3fcdd97463a8349915f5b672eba" member_type: VOTER last_known_addr { host: "127.6.192.61" port: 43865 } } peers { permanent_uuid: "bb30c63cba33444eba313d1530dc1b6e" member_type: VOTER last_known_addr { host: "127.6.192.60" port: 40075 } } }
I20260928 03:10:25.024519 30105 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72a9006384545468df49951fa7f2a1b [sys.catalog]: This master's current role is: LEADER
W20260928 03:10:25.279639 29913 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 03:10:25.277290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35390 (local address 127.6.192.61:43865)
0928 03:10:25.277575 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:25.277583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:25.278033 (+ 450us) server_negotiation.cc:407] Connection header received
0928 03:10:25.278227 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:25.278231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:25.278353 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:25.278447 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 03:10:25.279370 (+ 923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35390: BlockingRecv error: recv got EOF from 127.0.0.1:35390 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 03:10:25.413173 30025 catalog_manager.cc:1259] Loaded cluster ID: 2c04cf56b8484e7e8661700067c806b3
I20260928 03:10:25.413335 30025 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e: loading cluster ID for follower catalog manager: success
I20260928 03:10:25.416399 30025 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e: acquiring CA information for follower catalog manager: success
I20260928 03:10:25.418529 30025 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bb30c63cba33444eba313d1530dc1b6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 03:10:25.632599 30052 catalog_manager.cc:1259] Loaded cluster ID: 2c04cf56b8484e7e8661700067c806b3
I20260928 03:10:25.632721 30052 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba: loading cluster ID for follower catalog manager: success
I20260928 03:10:25.635607 30052 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba: acquiring CA information for follower catalog manager: success
I20260928 03:10:25.637676 30052 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f001c3fcdd97463a8349915f5b672eba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 03:10:25.804626 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:25.797516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35392 (local address 127.6.192.61:43865)
0928 03:10:25.798333 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:25.798345 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:25.798362 (+ 17us) server_negotiation.cc:407] Connection header received
0928 03:10:25.798474 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:25.798479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:25.798666 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:25.798821 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:10:25.799293 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:25.800332 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:25.801979 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:25.802488 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:25.802646 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:25.803188 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:10:25.803948 (+ 760us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:10:25.804125 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:25.804226 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 03:10:25.804320 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":488,"threads_started":1}
I20260928 03:10:25.820041 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:25.812593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40664 (local address 127.6.192.60:40075)
0928 03:10:25.813140 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:25.813146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:25.813447 (+ 301us) server_negotiation.cc:407] Connection header received
0928 03:10:25.813712 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:25.813720 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:25.813994 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:25.814148 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:10:25.814596 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:25.815357 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:25.817309 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:25.817723 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:25.817921 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:25.818539 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:10:25.819115 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:10:25.819237 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:25.819500 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 03:10:25.819597 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":267,"threads_started":1}
I20260928 03:10:25.854276 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:25.837610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40672 (local address 127.6.192.60:40075)
0928 03:10:25.837910 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:25.837921 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:25.844428 (+ 6507us) server_negotiation.cc:407] Connection header received
0928 03:10:25.844801 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:25.844807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:25.844943 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:25.845025 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:10:25.846179 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:25.849397 (+ 3218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:25.852102 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:25.852776 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:25.852874 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:25.853502 (+ 628us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:10:25.853634 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:10:25.853693 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:25.853903 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 03:10:25.853977 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 03:10:25.854276 30129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:25.837294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56118 (local address 127.6.192.62:46039)
0928 03:10:25.838737 (+ 1443us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:25.838747 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:25.844611 (+ 5864us) server_negotiation.cc:407] Connection header received
0928 03:10:25.845127 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:25.845133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:25.845372 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:25.845531 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:10:25.846447 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:25.847937 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:25.851272 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:25.851995 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:25.852097 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:25.852973 (+ 876us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:10:25.853492 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:10:25.853567 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:25.853834 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 03:10:25.853932 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1257,"thread_start_us":804,"threads_started":1}
I20260928 03:10:25.858753 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 03:10:25.844538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35402 (local address 127.6.192.61:43865)
0928 03:10:25.844762 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 03:10:25.844779 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 03:10:25.851208 (+ 6429us) server_negotiation.cc:407] Connection header received
0928 03:10:25.851578 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 03:10:25.851589 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 03:10:25.851703 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 03:10:25.851769 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 03:10:25.852708 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:25.853888 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:25.856407 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 03:10:25.856965 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 03:10:25.857107 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 03:10:25.857860 (+ 753us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 03:10:25.857986 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 03:10:25.858085 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 03:10:25.858301 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 03:10:25.858410 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 03:10:25.859764 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 29766
I20260928 03:10:25.860767 29828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:10:25.974313 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 29766
W20260928 03:10:26.004194 29934 connection.cc:535] server connection from 127.0.0.1:40648 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 03:10:26.004770 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 29840
I20260928 03:10:26.006068 29902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:10:26.125782 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 29840
I20260928 03:10:26.143653 6912 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task97hosG/build/asan/bin/kudu pid 29915
I20260928 03:10:26.144945 29977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 03:10:26.251889 6912 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97hosG/build/asan/bin/kudu with pid 29915
2026-09-28T03:10:26Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5066 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9594 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (381572 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST